DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's Amendments filed on 08/19/2026 have been entered and made of record.
Status of Claims
Currently pending Claim(s):
Amended claim(s):
1-20
5
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks made in an amendment received on 08/19/2026.
In view of amendments filed on 08/19/2026 to the claim, the objection to claim 5 is withdrawn.
In view of applicant’s arguments, Remarks filed on 08/19/2026, with respect to claims 1-20 under 35 U.S.C 101, the arguments have been fully considered and are found to be not persuasive. The Applicant argues on page 10:
PNG
media_image1.png
320
678
media_image1.png
Greyscale
The Examiner respectfully disagrees. The claimed operations are a mental process. A person can observe a ROI across captured image frames and determine whether the ROI is of valid size, changes positions or has been present within a scene for a valid duration of time. The computational scoring is a task a human could do mentally by deciding how to have a machine perform further processing on an image scene. For example, a human could assign a “validity score” of 0 or 1 to the image scene and combine these scores to determine if the scene is valid or invalid for SDOF processing. The mere recitation that a computer is used to execute these steps does not take the limitations out of the mental process grouping. The pending claims do not require specific structure to process the image to determine if the scene is valid or not valid for SDOF processing.
On page 11 of Remarks, the Applicant argues:
PNG
media_image2.png
500
682
media_image2.png
Greyscale
The Examiner respectfully disagrees. As explained above, a human can mentally observe an image scene and determine if a ROI in the scene becomes too small (or too large) and/or begins to move into (or out of) the scene. The pending claim does not recite the steps of a user pressing an image capture button on the electronic device. Further, providing a device with a "portrait mode auto suggest" merely provides an indication that the scene is valid or not valid for SDOF processing and is insignificant extra-solution activity as no SDOF processing is actually being performed. The limitation of “providing an indication…” is no more than mere instructions to apply the exception using a generic computer component.
Thus, even when viewed as a whole, nothing in the claim adds significantly more (i.e., an inventive concept) to the abstract idea. The claims are ineligible, and accordingly, the rejection of claims 1-20 under 35 U.S.C. 101 is maintained.
In view of applicant’s arguments, Remarks filed on 08/19/2026, with respect to independent claims 1, 6, and 15 under 35 U.S.C. 103, the arguments have been fully considered and are found to be not persuasive. The Applicant argues on page 13:
PNG
media_image3.png
304
696
media_image3.png
Greyscale
The Examiner respectfully disagrees. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “determining whether the overall composition of ROIs tracked in a scene being streamed by an image capture device”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Further, Seely discloses at Column 6 [lines 47-56] "identify one or more sub-regions within the image that contain what are likely to be clipped background highlight pixels, so that only the pixels in such identified sub-regions may be subjected to special highlight recovery-related processing during the rendering of the synthetic SDOF effects described herein." while “a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image." at Column 8 [lines 45-51]. Thus, the global highlight recovery/boosting score on Column 8 [lines 45-51] equates to validity score because the score indicates whether, and to what extent, a highlight recovery and boosting operation during synthetic SDOF rendering is applied.
The claim does not recite performing synthetic SDOF image processing. Rather, the claim recites determining that the scene is “valid or invalid for synthetic SDOF image processing”.[emphasis added]. As such, Seely’s score provides an indication associated with applying a synthetic SDOF rendering operation to the image.
Finaly, the rejection does not rely on Seely to teach “any temporal aspect of the appearance and/or placement of ROIs in the scene over a span of time preceding (or following) the current image.” Rather, the examiner relies on Avidan to teach determining, for each of the one or more ROIs identified within the first image, a temporal validity score at Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth.”
Continuing on page 13 of Remarks, the Applicant argues:
PNG
media_image4.png
308
702
media_image4.png
Greyscale
The Examiner respectfully disagrees. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “a computationally/programmatically-determined indication that is displayed on the UI of the device, indicating that the current scene has been adjudged to be valid for synthetic SDOF processing”) are not recited in the rejected claim(s). The pending claims simply recites “determining…that the scene is valid” based on “a combination of the ROI validify scores”. There is no mention of UI of the device or that a scene has been adjusted. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Finally on page 13 of Remarks, the Applicant argues:
PNG
media_image5.png
250
678
media_image5.png
Greyscale
The Examiner respectfully disagrees. The Applicant’s argument does not address the actual reasoning of the Examiner’s rejections. Instead, the Applicant attacks the references singly for lacking teachings that the Examiner relied on a combination of references to show. It is well established that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references (See In re Keller, 642 F.2d 413). The court requires that references must be read, not in isolation, but for what they fairly teach in combination with the prior art as a whole (See In re Keller, 642 F.2d 413, 425 (CCPA 1981); In re Merck & Co., 800 F.2d 1091 (Fed. Cir. 1986)).
Further, In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “analyzing a single camera’s captured image stream”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
As such, Seely was relied on to teach determining that the scene is “valid or invalid for synthetic SDOF image processing”, as explained above, whereas Avidan was relied on to teach determining, for each of the one or more ROIs identified within the first image, a temporal validity score at Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth.” Therefore, the combination of these two references meets the limitation of the pending claims.
Accordingly, the rejection of claims 1-20 under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion). The independent claims 1, 6, and 15 recite a method, a device, and a non-transitory computer readable medium. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that the independent claims 1, 6, and 15 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories? YES. Independent claims 1, 6, and 15 are directed to a process, machine, and manufacture.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims are directed toward a mental process (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
Independent claims 1, 6, and 15 comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding independent claim 1: the limitations recite:
A method of digital image processing, comprising:
obtaining a first image of a scene from an image stream captured by a first image capture device of an electronic device (data gathering);
identifying one or more regions of interest (ROI) within the first image (mental process including observation and evaluation, and can be done mentally in the human mind);
determining an ROI validity score for each of the one or more ROIs identified within the first image (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (additional element);
determining, for each of the one or more ROIs identified within the first image, a temporal validity score (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each temporal validity scores comprises an indication of whether the respective ROI is valid or invalid for synthetic SDOF image processing (additional element); and
determining, based, at least in part, on a combination of the ROI validity scores determined for each of the one or more ROIs identified within the first image and their respective temporal validity scores, that the scene is valid for synthetic SDOF image processing (mental process including observation and evaluation, and can be done mentally in the human mind); and
providing an indication to the electronic device that the scene is valid for synthetic SDOF image processing (extra solution activity).
Regarding independent claim 6: the limitations recite:
An electronic device (generic computer component), comprising:
a memory (generic computer component);
a first image capture device (generic computer component),; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to (generic computer component),:
obtain a first image of a scene from an image stream captured by a first image capture device of an electronic device (data gathering);
identify one or more regions of interest (ROI) within the first image (mental process including observation and evaluation, and can be done mentally in the human mind);
determine an ROI validity score for each of the one or more ROIs identified within the first image (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (additional element);
determine, for each of the one or more ROIs identified within the first image, a temporal validity score (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each temporal validity scores comprises an indication of whether the respective ROI is valid or invalid for synthetic SDOF image processing (additional element); and
determine, based, at least in part, on a combination of the ROI validity scores determined for each of the one or more ROIs identified within the first image and their respective temporal validity scores, that the scene is valid for synthetic SDOF image processing (mental process including observation and evaluation, and can be done mentally in the human mind); and
provide an indication to the electronic device that the scene is valid for synthetic SDOF image processing (extra solution activity).
Regarding independent claim 15: the limitations recite:
A non-transitory computer readable medium comprising computer readable instructions executable by one or more processors to (generic computer component):
obtain a first image of a scene from an image stream captured by a first image capture device of an electronic device (data gathering);
identify one or more regions of interest (ROI) within the first image (mental process including observation and evaluation, and can be done mentally in the human mind);
determine an ROI validity score for each of the one or more ROIs identified within the first image (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (additional element);
determine, for each of the one or more ROIs identified within the first image, a temporal validity score (mental process including observation and evaluation, and can be done mentally in the human mind), wherein each temporal validity scores comprises an indication of whether the respective ROI is valid or invalid for synthetic SDOF image processing (additional element); and
determine, based, at least in part, on a combination of the ROI validity scores determined for each of the one or more ROIs identified within the first image and their respective temporal validity scores, that the scene is valid for synthetic SDOF image processing (mental process including observation and evaluation, and can be done mentally in the human mind); and
provide an indication to the electronic device that the scene is valid for synthetic SDOF image processing (extra solution activity).
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
Paragraphs [0016] and [0017] describe the validity scores as estimations of positional validity, size validity, depth contrast validity (exemplified by an estimated difference in depth with respect to background) and an estimation of how much position has changed over a number of previous frames. The scores are used to determine whether some processing is suitable. The description does not appear to show any tasks that are specifically aimed at machine implementation. The scoring by estimation upon examination of video images is a task that a human could do mentally by deciding how to have a machine perform further processing on an image scene. The mere nominal recitation that the various steps are being executed by the generic computer components, for example, processor, device, memory, non-transitory computer readable medium etc. does not take the limitations out of the mental process grouping. Thus, the claims recite a mental process.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Independent claims 1, 6, and 15 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application. Independent claims 1, 6, and 15 discloses an processor, device, memory, non-transitory computer readable medium, obtaining a first image of a scene from an image stream captured by a first image capture device of an electronic device, wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing, wherein each temporal validity scores comprises an indication of whether the respective ROI is valid or invalid for synthetic SDOF image processing, and providing an indication to the electronic device that the scene is valid for synthetic SDOF image processing which are generic computer components and/or insignificant pre/post-solution extra activity that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a method.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Independent claim(s) 1, 6, and 15 do not recite any additional elements that are not well-understood, routine or conventional. The use of a generic computer elements are routine, well-understood and conventional process that is performed by computers.
Thus, since independent claims 1, 6, and 15 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that independent claims 1, 6, and 15 are not eligible subject matter under 35 U.S.C 101.
Regarding claims 2-5, 7-14, and 16-20: the additional elements do not integrate the mental processing into practical application or add significantly more to the mental process. In detail claims 2-5 depend on claim 1, claims 7-14 depend on claim 6, and claims 16-20 depend on claim 15.
apply a hysteresis factor, which comprises a certain number of images that have a scene that is valid or invalid or a duration of time which has been determined to have a scene that is valid or invalid (claims 2 and 3).
determine the number of frames and the duration of time is different from when it is determined the scene is valid or invalid (claim 4).
capture a subsequent image with data asset needed for the processing (claim 5).
wherein the ROI comprises a coordinate, a dimension, and a tracker identifier (claims 7 and 8).
using a trained machine leaning model to identify the ROI (claims 9 and 10).
wherein the validity score comprises positional component, size component, and depth component (claims 11 and 12)
wherein the temporal validity score indicates how much the ROI has moved and comprises a moving score based on that indication (claims 13 and 14).
wherein the temporal validity score comprises a movement threshold, which is proportional to the size of the ROI, and if the threshold is exceeded then the scene is invalid (claims 16 and 17).
determine the ROI has a valid validity and temporal validity score, wherein the score indicates how suited the scene is for processing and storing the score as metadata (claims 18-20).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-8, 11-15, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Seely et al. (US 10,992,845 B1) (hereinafter, “Seely”) in view of Avidan et al. (US 2009/0092282 A1) (hereinafter, “Avidan”).
Regarding claim 1, Seely discloses a method of digital image processing, comprising:
obtaining a first image of a scene from an image stream captured by a first image capture device of an electronic device (Column 6 [lines 15-17] "The scene 105 comprises human subject 115 in the foreground of scene 105, as well as several elements in the background of scene 105, e.g.: trees");
identifying one or more regions of interest (ROI) (sub-regions on Column 6 [lines 47-56] equate to ROI) within the first image (Column 6 [lines 47-56] "identify one or more sub-regions within the image that contain what are likely to be clipped background highlight pixels, so that only the pixels in such identified sub-regions may be subjected to special highlight recovery-related processing during the rendering of the synthetic SDOF effects described herein.");
determining an ROI validity score (global highlight recovery/boosting score on Column 8 [lines 38-44] equates to the validity score) for each of the one or more ROIs identified within the first image, wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”);
[determining, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication] of whether the respective ROI is valid or invalid for synthetic SDOF image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”); and
determining, based, at least in part, on a combination of the ROI validity scores (global highlight recovery/boosting score on Column 8 [lines 45-51] equates to validity score) determined for each of the one or more ROIs identified within the first image [and their respective temporal validity scores], that the scene is valid for synthetic SDOF image processing (Column 3 [lines 6-12] "the camera devices may utilize one (or more) image sensors to capture an input image"; Column 8 [lines 45-51] "A global highlight recovery/boosting score value of zero would indicate that no highlight recovery or boosting should be performed on the current image (e.g., perhaps due to their being too many pixels that are estimated to be clipped background pixels), while a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”; Column 24 [lines 46-48] " image capture device 950, which may, e.g., comprise single or multiple camera units/optical sensors having different characteristics"); and
providing an indication to the electronic device that the scene is valid for synthetic SDOF image processing (Column 24 [lines 59-65] "User interface 915 could, for example, be the conduit through which a user may view a captured video stream and/or indicate particular frame(s) that the user would like to have a synthetic SDOF image version of (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device's display screen).").
However, Seely fails to teach determining, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing].
Avidan teaches determining, for each of the one or more ROIs identified within the first image, a temporal validity score (match score on Paragraph [0032] equates to temporal validity score), wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing] (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth. We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include determining, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image] processing taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 1.
Regarding claim 6, Seely discloses an electronic device (Column 24 [lines 38-41] “Electronic device 900 could be, for example, a mobile telephone, personal media device, portable camera, or a tablet, notebook or desktop computer system.”, comprising:
a memory (Column 24 lines 41-50] “electronic device 900 may include…image capture device 950, which may, e.g., comprise single or multiple camera units/optical sensors having different characteristics, video codec(s) 955, memory 960, storage 965, and communications bus 970.”);
a first image capture device (Column 24 lines 41-50] “electronic device 900 may include…image capture device 950, which may, e.g., comprise single or multiple camera units/optical sensors having different characteristics, video codec(s) 955, memory 960, storage 965, and communications bus 970.”); and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to: (Column 24 [lines 51-56] “Processor 905 may execute instructions necessary to carry out or control the operation of many functions performed by device 900 (e.g., such as the generation and/or processing of images in accordance with the various embodiments described herein).)
obtain a first image of a scene from an image stream captured by a first image capture device of an electronic device (Column 6 [lines 15-17] "The scene 105 comprises human subject 115 in the foreground of scene 105, as well as several elements in the background of scene 105, e.g.: trees");
identify one or more regions of interest (ROI) (sub-regions on Column 6 [lines 47-56] equate to ROI) within the first image (Column 6 [lines 47-56] "identify one or more sub-regions within the image that contain what are likely to be clipped background highlight pixels, so that only the pixels in such identified sub-regions may be subjected to special highlight recovery-related processing during the rendering of the synthetic SDOF effects described herein.");
determine an ROI validity score (global highlight recovery/boosting score on Column 8 [lines 38-44] equates to the validity score) for each of the one or more ROIs identified within the first image, wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”);
[determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication] of whether the respective ROI is valid or invalid for synthetic SDOF image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”); and
determine, based, at least in part, on a combination of the ROI validity scores (global highlight recovery/boosting score on Column 8 [lines 45-51] equates to validity score) determined for each of the one or more ROIs identified within the first image [and their respective temporal validity scores], that the scene is valid for synthetic SDOF image processing (Column 3 [lines 6-12] "the camera devices may utilize one (or more) image sensors to capture an input image, as well as corresponding depth/disparity information data for the captured scene that may indicate the portions of the image that are in the scene's background and/or foreground (e.g., by using a secondary stereo camera and/or other types of depth sensors)."; Column 8 [lines 45-51] "A global highlight recovery/boosting score value of zero would indicate that no highlight recovery or boosting should be performed on the current image (e.g., perhaps due to their being too many pixels that are estimated to be clipped background pixels), while a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”; Column 24 [lines 46-48] " image capture device 950, which may, e.g., comprise single or multiple camera units/optical sensors having different characteristics"); and
provide an indication to the electronic device that the scene is valid for synthetic SDOF image processing (Column 24 [lines 59-65] "User interface 915 could, for example, be the conduit through which a user may view a captured video stream and/or indicate particular frame(s) that the user would like to have a synthetic SDOF image version of (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device's display screen).").
However, Seely fails to teach determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing].
Avidan teaches determine, for each of the one or more ROIs identified within the first image, a temporal validity score (match score on Paragraph [0032] equates to temporal validity score), wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing] (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth. We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing] taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 6.
Regarding claim 7, which claim 6 is incorporated, Seely discloses wherein each ROI comprises: at least one coordinate within the first image; and at least one dimension (Column 9 [lines 23-37] "the first stage of a highlight recovery process may be to perform a scan of the image and constructing a tile processing list (e.g., as represented graphically by tile processing list 185 shown in FIG. 1C) that contains the positions of the tiles that need to be processed for highlight recovery...region 182 consists of four individual tiles 190, each consisting of a set of 64 pixels laid out in an 8 pixel by 8 pixel square.").
Regarding claim 8, which claim 6 is incorporated, Seely fails to teach wherein each ROI comprises a tracking identifier, and wherein the tracking identifier for a respective ROI remains the same for as long as the respective ROI remains identified in the captured images from the image stream.
Avidan teaches wherein each ROI comprises a tracking identifier, and wherein the tracking identifier for a respective ROI remains the same for as long as the respective ROI remains identified in the captured images from the image stream (Paragraph [0015] "An initial 2D tracking window specifies the 2D (x, y) location of the object to be tracked in the first image. This can be done manually or by some conventional object detection method. An initial depth d of the object is determined using a conventional plane-sweep procedure. The 2D (x, y) location and depth d, in combination indicate the 3D position of the object.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include wherein each ROI comprises a tracking identifier, and wherein the tracking identifier for a respective ROI remains the same for as long as the respective ROI remains identified in the captured images from the image stream taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 8.
Regarding claim 11, which claim 6 is incorporated, Seely discloses wherein at least one ROI validity score comprises at least: a positional validity component; and a size validity component (Column 9 [lines 23-37] "the first stage of a highlight recovery process may be to perform a scan of the image and constructing a tile processing list (e.g., as represented graphically by tile processing list 185 shown in FIG. 1C) that contains the positions of the tiles that need to be processed for highlight recovery...region 182 consists of four individual tiles 190, each consisting of a set of 64 pixels laid out in an 8 pixel by 8 pixel square.").
Regarding claim 12, which claim 6 is incorporated, Seely discloses wherein at least one ROI validity score comprises a depth contrast component, and wherein the depth contrast component indicates a difference in an estimated depth of the respective ROI and an estimated depth of a background of the first image (Column 11 [lines 28-44] "depth and/or disparity estimates 218 may be obtained for the captured image 216, e.g., via the use of defocus information, phase detection pixels, or other desired modalities. According to some embodiments, semantic segmentation information 220 may also be obtained for captured image 216, e.g., in the form of a segmentation mask providing a per-pixel classification of pixels in the image (such as ‘foreground’ vs. ‘background,’ or ‘person’ vs. ‘non-person,’ etc.)...values reflecting the relative probabilities that a pixel belongs to a particular segmented class. In some cases, as indicated by the dashed line arrow in FIG. 2A, the depth and/or disparity estimates 218 may be used to further refine the semantic segmentation information 220.").
Regarding claim 13, which claim 6 is incorporated, Seely fails to teach wherein each temporal validity score comprises an indication of how much a position of the respective ROI has moved over a number of previously-captured images from the image stream.
Avidan teaches wherein each temporal validity score comprises an indication of how much a position of the respective ROI has moved over a number of previously-captured images from the image stream (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score...We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include wherein each temporal validity score comprises an indication of how much a position of the respective ROI has moved over a number of previously-captured images from the image stream taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 13.
Regarding claim 14, which claim 13 is incorporated, Seely fails to teach wherein each temporal validity score comprises a moving score value based on the position of the respective ROI over the number of previously-captured images from the image stream.
Avidan teaches wherein each temporal validity score comprises a moving score value based on the position of the respective ROI over the number of previously-captured images from the image stream (Paragraph [0022] "Then, we perform matching between the synthetic aperture images for windows centered near the 2D location of the object in the previous image and select the 3D positions of the object that matches best, i.e., a particular 2D location and a particular depth with the maximal score."; Paragraph [0032] “for each next set of images 101 from j=1 to k, determine 110 the synthetic aperture images 115 for a range of depths centered near the depth of the previous set of images. For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include wherein each temporal validity score comprises a moving score value based on the position of the respective ROI over the number of previously-captured images from the image stream taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 14.
Regarding claim 15, Seely discloses a non-transitory computer readable medium comprising computer readable instructions executable by one or more processors to (Column 25 [lines 42-48] “Memory 960 and storage 965 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 905 such computer program code may implement one or more of the methods described herein.”):
obtain a first image of a scene from an image stream captured by a first image capture device of an electronic device (Column 6 [lines 15-17] "The scene 105 comprises human subject 115 in the foreground of scene 105, as well as several elements in the background of scene 105, e.g.: trees");
identify one or more regions of interest (ROI) (sub-regions on Column 6 [lines 47-56] equate to ROI) within the first image (Column 6 [lines 47-56] "identify one or more sub-regions within the image that contain what are likely to be clipped background highlight pixels, so that only the pixels in such identified sub-regions may be subjected to special highlight recovery-related processing during the rendering of the synthetic SDOF effects described herein.");
determine an ROI validity score (global highlight recovery/boosting score on Column 8 [lines 38-44] equates to the validity score) for each of the one or more ROIs identified within the first image, wherein each ROI validity score comprises an indication of whether the scene is valid or invalid for synthetic shallow depth of field (SDOF) image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined." Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”;);
[determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication] of whether the respective ROI is valid or invalid for synthetic SDOF image processing (Column 8 [lines 38-44] "by applying another soft threshold (or other desired form of threshold) to the percentage of clipped background pixels and the exposure score, and then taking the product of said soft-thresholded clipped background pixels percentage and exposure score, a global highlight recovery/boosting score for the image may be determined."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”); and
determine, based, at least in part, on a combination of the ROI validity scores (global highlight recovery/boosting score on Column 8 [lines 45-51] equates to validity score) determined for each of the one or more ROIs identified within the first image [and their respective temporal validity scores,] that the scene is valid for synthetic SDOF image processing (Column 3 [lines 6-12] "the camera devices may utilize one (or more) image sensors to capture an input image"; Column 8 [lines 45-51] "A global highlight recovery/boosting score value of zero would indicate that no highlight recovery or boosting should be performed on the current image (e.g., perhaps due to their being too many pixels that are estimated to be clipped background pixels), while a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”; Column 24 [lines 46-48] " image capture device 950, which may, e.g., comprise single or multiple camera units/optical sensors having different characteristics"); and
provide an indication to the electronic device that the scene is valid for synthetic SDOF image processing (Column 24 [lines 59-65] "User interface 915 could, for example, be the conduit through which a user may view a captured video stream and/or indicate particular frame(s) that the user would like to have a synthetic SDOF image version of (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device's display screen).").
However, Seely fails to teach determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing].
Avidan teaches determine, for each of the one or more ROIs identified within the first image, a temporal validity score (match score on Paragraph [0032] equates to temporal validity score), wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing] (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth. We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include determine, for each of the one or more ROIs identified within the first image, a temporal validity score, wherein each temporal validity scores comprises an indication [of whether the respective ROI is valid or invalid for synthetic SDOF image processing] taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 15.
Regarding claim 18, which claim 15 is incorporated, Seely discloses wherein the computer readable instructions executable by one or more processors (Column 25 [lines 42-48] “Memory 960 and storage 965 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 905 such computer program code may implement one or more of the methods described herein.”) to determine that the scene is valid for synthetic SDOF image processing further comprise computer readable instructions executable by one or more processors to: determine that at least one identified ROI has a valid ROI validity score (global highlight recovery/boosting score on Column 8 [lines 45-51] equates to validity score) (Column 8 [lines 45-51] "A global highlight recovery/boosting score value of zero would indicate that no highlight recovery or boosting should be performed on the current image (e.g., perhaps due to their being too many pixels that are estimated to be clipped background pixels), while a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image."; Column 10 [lines 35-50] “the SDOF rendering process may involve a sparse sampling process, wherein a number of pixels surrounding each pixel in the output image are sampled…the contribution weight of a sampled pixel and the sampled pixel's color value may each be further modified, e.g., according to a boosting score…the boosting score may be multiplied by a pair of gain coefficients (referred to herein as a weight gain coefficient and an intensity gain coefficient), which may be determined, at least in part, by the gating stage above, e.g., by multiplying the global highlight recovery/boosting score for the image”) and [a valid temporal validity score].
However, Seely fails to teach a valid temporal validity score.
Avidan teaches a valid temporal validity score (match score on Paragraph [0032] equates to temporal validity score) (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth. We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include a valid temporal validity score taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan with Seely to obtain the invention specified in claim 18.
Regarding claim 19, which claim 15 is incorporated, Seely discloses further comprising computer readable instructions executable by one or more processors to (Column 25 [lines 42-48] “Memory 960 and storage 965 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 905 such computer program code may implement one or more of the methods described herein.”): to determine at least one score for the first image, wherein the at least one score is indicative of how well-suited the scene is for synthetic SDOF image processing (Column 8 [lines 45-51] "A global highlight recovery/boosting score value of zero would indicate that no highlight recovery or boosting should be performed on the current image (e.g., perhaps due to their being too many pixels that are estimated to be clipped background pixels), while a value of one would indicate that full-strength highlight recovery and boosting should be applied to the image."; Column 11 [lines 55-58] “the highlight recovery gating information 222 may take the form of a global indication of whether or not highlight recovery should be performed on the image if subjected to a subsequent synthetic SDOF rendering process”; Column 12 [lines 21-28] ""depth and/or disparity estimates 218, semantic segmentation information 220, highlight recovery gating information 222…, and dynamically-tuned parameters 224 may be stored, along with the captured image 216, e.g., in the form of metadata information, as image 226the input image to the render stage of the image processing pipeline").
Regarding claim 20, which claim 19 is incorporated, Seely discloses computer readable instructions executable by one or more processors to (Column 25 [lines 42-48] “Memory 960 and storage 965 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 905 such computer program code may implement one or more of the methods described herein.”): store the at least one score in metadata of an image from the image stream captured subsequently to the capture of the first image (Column 8 [lines [1-5] " the portions of the image processing pipeline 200 illustrated in FIG. 2A are typically configured to occur at capture time (or subsequently thereafter), i.e., in an image signal processor or related firmware or software.").
Claims 2-5 are rejected under 35 U.S.C. 103 as being unpatentable over Seely et al. (US 10,992,845 B1) (hereinafter, “Seely”) in view of Avidan et al. (US 2009/0092282 A1) (hereinafter, “Avidan”), and further in view of Poplin (US 6,943,832 B2).
Regarding claim 2, which claim 1 is incorporated, Seely discloses [applying a hysteresis factor to the] providing of the indication that the scene is valid for synthetic SDOF image processing (Column 24 [lines 59-65] "User interface 915 could, for example, be the conduit through which a user may view a captured video stream and/or indicate particular frame(s) that the user would like to have a synthetic SDOF image version of (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device's display screen).").
However, Seely and Avidan both fail to teach applying a hysteresis factor.
Poplin teaches applying a hysteresis factor (Column 6 [lines 43-44] "method 200 for applying hysteresis to an image control feature”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include applying a hysteresis factor taught by Poplin’s reference. The motivation for doing so would have been to determine if an image has changed enough to warrant an adjustment as suggested by Poplin (see Poplin, end of column 1 continued on to first line of column 2).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Poplin with Seely to obtain the invention specified in claim 2.
Regarding claim 3, which claim 2 is incorporated, Seely discloses synthetic SDOF (Column 24 [lines 59-65] "User interface 915 could, for example, be the conduit through which a user may view a captured video stream and/or indicate particular frame(s) that the user would like to have a synthetic SDOF image version of (e.g., by clicking on a physical or virtual button at the moment the desired frame is being displayed on the device's display screen).").
However, Seely and Avidan both fail to teach wherein the hysteresis factor comprises at least one of: (a) a number of consecutive images captured from the image stream prior to the first image that must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (b) a duration of time over which consecutive images captured from the image stream prior to the first image must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (c) a number of consecutive images captured from the image stream subsequently to the first image that must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication; or (d) a duration of time over which consecutive images captured from the image stream subsequently to the first image must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication.
Poplin teaches wherein the hysteresis factor comprises at least one of (Column 6 [lines 43-44] "method 200 for applying hysteresis to an image control feature”):
(a) a number of consecutive images captured from the image stream prior to the first image that must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication (Examiner notes that claims requires at least one of (a) a number of consecutive images captured from the image stream prior to the first image that must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (b) a duration of time over which consecutive images captured from the image stream prior to the first image must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (c) a number of consecutive images captured from the image stream subsequently to the first image that must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication; or (d) a duration of time over which consecutive images captured from the image stream subsequently to the first image must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication);
(b) a duration of time over which consecutive images captured from the image stream prior to the first image must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication (Column 10 [lines 22-33] “In the acquisition mode, the AE image processing algorithm adjusts the exposure, for example, image frame-by-image frame until the luminance is 20% of full scale to within some relatively narrow tolerance. Once this result is achieved, the AE image processing algorithm switches to maintenance mode. In maintenance mode, the AE image processing algorithm continues to monitor the luminance, but does not change or adjust the exposure as long as the luminance remains within some relatively wide tolerance. If the luminance goes outside the wide tolerance, the AE image processing algorithm switches back to acquisition mode.);
(c) a number of consecutive images captured from the image stream subsequently to the first image that must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication (Examiner notes that claims requires at least one of (a) a number of consecutive images captured from the image stream prior to the first image that must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (b) a duration of time over which consecutive images captured from the image stream prior to the first image must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (c) a number of consecutive images captured from the image stream subsequently to the first image that must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication; or (d) a duration of time over which consecutive images captured from the image stream subsequently to the first image must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication); or
(d) a duration of time over which consecutive images captured from the image stream subsequently to the first image must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication (Column 10 [lines 27-33] “ In maintenance mode, the AE image processing algorithm continues to monitor the luminance, but does not change or adjust the exposure as long as the luminance remains within some relatively wide tolerance. If the luminance goes outside the wide tolerance, the AE image processing algorithm switches back to acquisition mode.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include wherein the hysteresis factor comprises at least one of: (a) a number of consecutive images captured from the image stream prior to the first image that must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (b) a duration of time over which consecutive images captured from the image stream prior to the first image must have been determined to have a scene that is valid for [synthetic SDOF] processing before providing the indication; (c) a number of consecutive images captured from the image stream subsequently to the first image that must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication; or (d) a duration of time over which consecutive images captured from the image stream subsequently to the first image must have been determined to have a scene that is invalid for [synthetic SDOF] processing before removing the indication taught by Poplin’s reference. The motivation for doing so would have been to determine if an image has changed enough to warrant an adjustment as suggested by Poplin (see Poplin, end of column 1 continued on to first line of column 2).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Poplin with Seely to obtain the invention specified in claim 3.
Regarding claim 4, which claim 3 is incorporated, Seely and Avidan both fail to teach (1) the number of consecutive images in (a) is different than the number of consecutive images in (c); or (2) the duration of time in (b) is different than the duration of time in (d).
Poplin teaches (1) the number of consecutive images in (a) is different than the number of consecutive images in (c) (Examiner notes that claim requires only one of the limitations of (1) the number of consecutive images in (a) is different than the number of consecutive images in (c); or (2) the duration of time in (b) is different than the duration of time in (d)); or (2) the duration of time in (b) is different than the duration of time in (d) (Column 3 line 64 continued to column 4 line “ an image processing algorithm with hysteresis makes an adjustment by measuring a parameter and comparing it to a target value. Hysteresis is added to or incorporated into the image processing algorithm by defining a relatively narrow tolerance for an acquisition mode and a somewhat wider tolerance for a maintenance mode. The magnitude of the tolerances may change from one algorithm (i.e., an image processing algorithm for AE) to another algorithm (i.e., an image processing algorithm for AWB)”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include (1) the number of consecutive images in (a) is different than the number of consecutive images in (c); or (2) the duration of time in (b) is different than the duration of time in (d) taught by Poplin’s reference. The motivation for doing so would have been to determine if an image has changed enough to warrant an adjustment as suggested by Poplin (see Poplin, end of column 1 continued on to first line of column 2).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Poplin with Seely to obtain the invention specified in claim 4.
Regarding claim 5, which claim 3 is incorporated, Seely discloses wherein, in response to providing the indication, the method further comprises: capturing, by the first image capture device, an image subsequently to the capture of the first image, wherein the subsequently captured image comprises at least one data asset needed for synthetic SDOF processing (Column 3 [lines 6-19] " the camera devices may utilize one (or more) image sensors to capture an input image, as well as corresponding depth/disparity information data for the captured scene ...the depth information data may be converted into and/or represented in the form of a blur map, e.g., a two-dimensional array of blur values, wherein each blur value represents a radius, diameter (or other size-indicative parameter) of the blurring “circle of confusion” of the corresponding pixel in the captured image."; Column 11 [lines 65-67] “a blur map 228 may also be generated from the captured image's depth/disparity information (or otherwise obtained) during the rendering stage of the SDOF image.”).
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Seely et al. (US 10,992,845 B1) in view of Avidan et al. (US 2009/0092282 A1), and further in view of Sun et al. (US 10,897,558 B1).
Regarding claim 9, which claim 6 is incorporated, Seely and Avidan both fail to teach wherein at least one ROI is identified based, at least in part, on an output of a Machine Learning (ML)-based model used to analyze the first image.
Sun teaches teach wherein at least one ROI is identified based, at least in part, on an output of a Machine Learning (ML)-based model used to analyze the first image (Column 13 [lines 27-30] "The segmentation masks obtained at Step 206 may comprise binary or continuous segmentation masks. The segmentation masks and corresponding confidence masks may be produced by a neural network or other machine learning-based system...The confidence masks may reflect the relative confidence that the segmentation operation has in its classification of any given pixel in the reference color image.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include wherein at least one ROI is identified based, at least in part, on an output of a Machine Learning (ML)-based model used to analyze the first image taught by Sun’s references. The motivation for doing so would have been to produce a segmentation mask and the corresponding confidence mask of any given pixel in the reference image as suggested by Sun (see Sun, Column 13 [lines 32-35]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Sun with Seely to obtain the invention specified in claim 9.
Regarding claim 10, which claim 9 is incorporated, Seely and Avidan both fail to teach wherein the ML-based model is trained to recognize one or more salient objects in an analyzed image. Sun teaches wherein the ML-based model is trained to recognize one or more salient objects in an analyzed image (Column 13 [lines 27-30] "The segmentation masks obtained at Step 206 may comprise binary or continuous segmentation masks. The segmentation masks and corresponding confidence masks may be produced by a neural network or other machine learning-based system...The confidence masks may reflect the relative confidence that the segmentation operation has in its classification of any given pixel in the reference color image.").
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include wherein the ML-based model is trained to recognize one or more salient objects in an analyzed image taught by Sun’s references. The motivation for doing so would have been to produce a segmentation mask and the corresponding confidence mask of any given pixel in the reference image as suggested by Sun (see Sun, Column 13 [lines 32-35]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Sun with Seely to obtain the invention specified in claim 10.
Claims 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Seely et al. (US 10,992,845 B1) (hereinafter, “Seely”) in view of Avidan et al. (US 2009/0092282 A1) (hereinafter, “Avidan”), and further in view of Kurtz et al. (US 2010/0245532 A1) (hereinafter, “Kurtz”).
Regarding claim 16, which claim 15 is incorporated, Seely fails to teach wherein each temporal validity score comprises use of a movement threshold value that, if exceeded, results in an invalid temporal validity score for the respective ROI.
Avidan teaches the temporal validity score (Paragraph [0032] "For each of these images 115, we determine 120 the match scores 125 to the eigenspace for the 2D tracking windows. We set 130 the new positions (x, y, depth) 135 of the object to be the window with the maximal match score. We scale 140 the 2D window and the basis images 145 to account for any change in depth. We project 150 the scaled window to each camera to get N images of the target for the current image, and determine the variance 155 across these images. If the variance is lower than a predetermined threshold, then object is unoccluded and update 160 the eigenspace with the synthetic aperture image of the object. In any case, proceed by processing the next set of images.".
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely’s reference to include the temporal validity score taught by Avidan’s reference. The motivation for doing so would have been to perform matching between the images for windows centered near the 2D location of the object in the previous image and select the positions of the object that matches best as suggested by Avidan (see Avidan, Paragraph [0022]).
However, Seely and Avidan both fail to teach wherein each [temporal validity score] comprises use of a movement threshold value that, if exceeded, results in an invalid [temporal validity score] for the respective ROI.
Kurtz teaches teach wherein each [temporal validity score] comprises use of a movement threshold value that, if exceeded, results in an invalid [temporal validity score] for the respective ROI (Abstract “The subject movement is analyzed relative to the determined magnitude, direction of subject movement, and the current framing, to determine that the subject is properly framed by the current shot framing or whether modifications of the current shot framing are required to capture the moving subject, or to determine a new shot selection and new shot framing. The camera re-frames the subject if the subject is determined not to be properly framed”; Paragraph [0115] “User activity can also exceed thresholds for re-sizing…or level of motion thresholds for subject movement…or reframing frequency threshold…or camera adjustment limits, such that reframing becomes necessary and is then defined in a determine adjustment parameters step 935.”;) (Examiner interprets when the subject is determined not to be properly framed, this would correspond to an invalid indication for a ROI).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include wherein each [temporal validity score] comprises use of a movement threshold value that, if exceeded, results in an invalid [temporal validity score] for the respective ROI taught by Kurtz’s references. The motivation for doing so would have been to decrease viewer annoyance by adjusting the frame as the user moves suggested by Kurtz (see Kurtz, Paragraph [0115]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Kurtz with Seely to obtain the invention specified in claim 16.
Regarding claim 17, which claim 16 is incorporated, Seely and Avidan both fail to teach wherein the movement threshold value is proportional to a size of the respective ROI.
Kurtz teaches wherein the movement threshold value is proportional to a size of the respective ROI (Paragraph [0114] “As Table 3 shows, these time delays can be shot dependent. In the case of time delay…the tighter the shot, the more quickly the system will respond to reframe the subject. It is noted that if a subject 10 moves back into the frame center area 424 before a …time delay has passed, the anticipated reframing changes can be cancelled or changed. In moving from a current shot 428 to a new shot 428 on the basis of subject settling, the amount of time delay…after the subject 10 has been considered to settle, plus prior time spent in the current shot, should exceed a specified minimum shot durations Ts.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Seely in view of Avidan to include wherein the movement threshold value is proportional to a size of the respective ROI taught by Kurtz’s references. The motivation for doing so would have been to decrease viewer annoyance by adjusting the frame as the user moves suggested by Kurtz (see Kurtz, Paragraph [0115]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Avidan and Kurtz with Seely to obtain the invention specified in claim 17.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wilson et al. (US 2020/0074642 A1) discloses a method for motion assisted image segmentation that adjusts the foreground pixels by using a cumulative optical flow map between the first frame and the last frame of the plurality of frames.
Agrawal et al. (US 10,217,195 B1) discloses a method for image editing that uses segmentation image data may identify a first portion of the color image data as a foreground region and a second portion of the color image data as a background region to generate a blended and blurred representation of the first area and the second area.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
