Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification, as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f), except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f), except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
In claim 8, “an acquiring unit,” “an image segmenting unit,” “an effect processing unit,” and “an insert object selecting sub-unit.”
In dependent claims 10 and 12-14, each of the claimed units and subunits.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
A review of the specification shows the following structure corresponding to the claim limitations: Specification, fig 10, paragraph 115, lines 1-4, and paragraph 116, describe that the apparatus may be comprised of a processor executing instructions stored in memory, where the instructions implement the claimed functions.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 8-11, 13, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hold-Geoffroy et al. (U.S. Patent Application Publication No. 2018/0359416), referred herein as Hold, in view of Sytnik et al. (U.S. Patent Application Publication No. 2021/0150682), referred herein as Sytnik.
Regarding claim 1, Hold teaches an image processing method comprising: acquiring a first image, and acquiring attribute information of sky in the first image, the attribute information comprising at least one of weather, time, and type of region (fig 1; paragraphs 31 and 32; paragraph 34, the last 8 lines; paragraph 35, lines 1-2; paragraph 36, lines 7-12; paragraphs 48 and 56; attribute information of the sky is acquired, including weather, time, and/or type of region);
performing sky segmentation processing on the first image to obtain a sky image in the first image, wherein performing sky segmentation processing on the first image to obtain the sky image in the first image comprises identifying a ground boundary in the first image so as to use sky pixel lighting information above the ground boundary in the sky image (fig 3; paragraph 32, the last 6 lines; paragraph 42, lines 1-3; paragraph 43; paragraph 45, lines 1-8; paragraph 46, lines 1-2; paragraph 52, lines 1-7; paragraph 64, lines 1-11; an environment map that comprises an identified horizon [a ground boundary] and sky information above the horizon, is used when processing the sky image); and
performing corresponding effect processing on the sky image in the first image based on the attribute information; wherein the performing corresponding effect processing on the sky image in the first image based on the attribute information comprises: obtaining an insert object matched with the attribute information, and inserting the insert object to the sky image (fig 1, object 112; paragraphs 31 and 38; paragraph 64, lines 1-11; paragraph 105; effect processing on the sky image is performed based on the attribute information, comprising obtaining an insert object matched with the attribute information, and inserting the object into the sky image),
wherein the acquiring the attribute information of the sky in the first image comprises: acquiring at least one of time, type of region, and weather at a located position, when the first image is shot by a shooting device; and using at least one of the time, the type of region, and the weather as the attribute information of the sky (fig 3; paragraph 32, lines 1-7; paragraph 43; paragraph 48; paragraph 105, lines 1-6; the acquired time, type of region, and/or weather attributes used in the effects processing are from the time at which the image is captured by a camera).
Hold teaches identifying a ground boundary (e.g. a horizon) and using the sky information above the ground boundary to generate the sky image (please see the citations above). However, Hold does not explicitly teach using a pixel image above the ground boundary as the sky image.
However, in a similar field of endeavor, Sytnik teaches an image processing method comprising acquiring a first image and attribute information of sky in the first image, and performing sky segmentation processing on the first image to obtain a sky image in the first image (figs 17 and 20; paragraph 73, lines 16-19; paragraph 74; paragraph 77, lines 1-11 and the last 4 lines; paragraph 88, lines 1-4 and 14-25), wherein the sky segmentation comprises identifying a ground boundary so as to use a pixel image above the ground boundary as the sky image (figs 18 and 21-24; paragraph 70, lines 1-7 and the last 8 lines; paragraph 73, the last 2 lines; paragraph 74, lines 1-8; paragraph 78, lines 1-11; paragraph 88, lines 1-4 and 14-25).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the sky pixel image processing of Sytnik with the sky image processing of Hold because this helps to more efficiently perform the replacement and/or modification of the sky with improved accuracy, regardless of image complexity, which is especially important in sky image processing such as that disclosed by Hold (see, for example, Sytnik, paragraphs 3 and 4).
Regarding claim 2, Hold in view of Sytnik teaches the method according to claim 1, wherein the acquiring the attribute information of the sky in the first image comprises: processing the first image based on a preset identification model to obtain the attribute information of the sky in the first image (Hold, paragraph 32, lines 1-7; paragraph 36, lines 7-12; paragraph 48; paragraph 56; Sytnik, paragraph 73, lines 1-7; paragraph 107; the motivation to combine is the same as that discussed in the rejection of claim 1).
Regarding claim 3, Hold in view of Sytnik teaches the method according to claim 1, wherein the acquiring the attribute information of the sky in the first image further comprises: acquiring an orientation; and using the orientation as the attribute information of the sky (Hold, paragraph 31; paragraph 32, lines 1-7; paragraph 36, lines 7-12; paragraph 54; Sytnik, paragraph 95; the motivation to combine is the same as that discussed in the rejection of claim 1).
Regarding claim 4, Hold in view of Sytnik teaches the method according to claim 1, wherein the insert object comprises at least one of an image, animation or AR object (Hold, fig 1; paragraph 38, lines 1-13; paragraph 105; Sytnik, paragraphs 70, 89, 96; the motivation to combine is the same as that discussed in the rejection of claim 1).
Regarding claim 6, Hold in view of Sytnik teaches the method according to claim 4, wherein after inserting the insert object to the sky image, the method further comprises: regulating a brightness of the insert object according to a brightness of an insert position, so that the brightness of the insert object is matched with the brightness of the insert position (Hold, fig 1; paragraph 32; paragraph 34, the last 8 lines; paragraph 35, lines 1-2; paragraph 38, lines 1-13; Sytnik, paragraphs 96-98; the motivation to combine is the same as that discussed in the rejection of claim 1).
Regarding claim 8, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the structural components, which are taught by Hold, fig 12, processor 1202, memory 1204; paragraph 123); thus they are rejected on similar grounds.
Regarding claims 9-11 and 13, the limitations of these claims substantially correspond to the limitations of claims 2-4 and 6, respectively (structural components are taught by Hold, as discussed above in the rejection of claim 8); thus they are rejected on similar grounds as their corresponding claims.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 1 (except for the memory, processor, and program, which are taught by Hold, fig 12, memory 1204, processor 1202; paragraph 123); thus they are rejected on similar grounds.
Regarding claim 16, Hold in view of Sytnik teaches a non-transitory computer readable storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor (Hold, fig 12, medium 1204, processor 1202; paragraph 123), the method according to claim 1 is implemented (please see the rejection of claim 1, above).
Claims 5, 7, 12, 14, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hold, in view of Sytnik, and further in view of Izawa et al. (U.S. Patent Application Publication No. 2009/0087123), referred herein as Izawa.
Regarding claim 5, Hold in view of Sytnik teaches the method according to claim 4, wherein the determining the insert object matched with the attribute information based on the attribute information comprises: showing a candidate object matched with the attribute information based on the attribute information; and selecting the candidate object as the insert object, in response to an insertion operation on the candidate object (Hold, fig 1; paragraph 38, lines 1-13; paragraph 105; Sytnik, paragraphs 89 and 96-98; the motivation to combine is the same as that discussed in the rejection of claim 1).
Hold in view of Sytnik teaches providing a database of attributes (Hold, paragraph 96, lines 1-10; Sytnik, fig 3; the motivation to combine is the same as that discussed in the rejection of claim 1), but does not teach showing a plurality of candidate objects, and selecting a candidate object of the plurality of candidate objects as the insert object in response to a selection operation on the candidate object.
However, in a similar field of endeavor, Izawa teaches an image processing method comprising acquiring a first image, identifying sky regions in the first image, and performing effect processing on the sky region (paragraphs 101 and 103; paragraph 104, lines 1-7; paragraphs 105 and 106), wherein the method further comprises showing a plurality of candidate objects, and selecting a candidate object of the plurality of candidate objects as an insert object in response to a selection operation on the candidate object (fig 2, the plurality of candidate objects in the object image database; paragraphs 113 and 114; a plurality of objects are shown, and one of the objects is selected for insertion into the first image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the object selection of Izawa with the object insertion of Hold in view of Sytnik because this makes it possible for custom selection of an object for insertion into the image and allows for changes or corrections to be made as desired, thereby improving both the image processing process and the output image result (see, for example, Izawa, paragraph 115, the last 4 lines).
Regarding claim 7, Hold in view of Sytnik teaches the method according to claim 4, wherein the inserting the insert object to the sky image comprises: acquiring the insert position on the sky image, and inserting the insert object to the insert position on the sky image (Hold, fig 1; paragraph 38, lines 1-13; paragraph 105; Sytnik, paragraphs 88, 89, and 91; the motivation to combine is the same as that discussed in the rejection of claim 1).
Hold in view of Sytnik teaches facilitating user interaction with the image processing system through a user interface (Hold, paragraphs 96 and 126; Sytnik, paragraphs 82 and 89; paragraphs 95 and 97; the motivation to combine is the same as that discussed in the rejection of claim 1), but does not explicitly teach that the position is specified by a user.
However, in a similar field of endeavor, Izawa teaches an image processing method comprising acquiring a first image, identifying sky regions in the first image, and performing effect processing on the sky region (paragraphs 101 and 103; paragraph 104, lines 1-7; paragraphs 105 and 106), wherein the effect processing comprises inserting an object in the first image by acquiring an insert position specified by a user, and inserting the insert object to the insert position specified by the user (paragraphs 105 and 106; paragraph 114; paragraph 115, lines 1-5).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the user position selection of Izawa with the image insertion of Hold in view of Sytnik because this makes it possible for custom selection of an object for insertion into the image and allows for changes or corrections to be made as desired, thereby improving both the image processing process and the output image result (see, for example, Izawa, paragraph 115, the last 4 lines).
Regarding claims 12 and 14, the limitations of these claims substantially correspond to the limitations of claims 5 and 7, respectively; thus they are rejected on similar grounds as their corresponding claims.
Regarding claim 17, Hold in view of Sytnik, further in view of Izawa teaches the method according to claim 5, wherein after inserting the insert object to the sky image, the method further comprises: regulating a brightness of the insert object according to a brightness of an insert position, so that the brightness of the insert object is matched with the brightness of the insert position (Hold, fig 1; paragraph 32; paragraph 34, the last 8 lines; paragraph 35, lines 1-2; paragraph 38, lines 1-13; Sytnik, paragraphs 96-98; the motivation to combine is the same as that discussed in the rejection of claim 1; this subject matter substantially corresponds to that of claim 6).
Regarding claim 18, Hold in view of Sytnik, further in view of Izawa teaches the method according to claim 5, wherein the inserting the insert object to the sky image comprises: acquiring the insert position on the sky image specified by a user; and inserting the insert object to the insert position on the sky image specified by the user (Hold, fig 1; paragraph 38, lines 1-13; paragraph 105; Sytnik, paragraphs 88, 89, and 91; paragraphs 82, 89, 95, and 97; the motivation to combine is the same as that discussed in the rejection of claim 1; Izawa, paragraphs 105 and 106; paragraph 114; paragraph 115, lines 1-5; this subject matter substantially corresponds to that of claim 7; the motivation to combine is the same as that discussed in the rejection of claim 7).
Regarding claims 19 and 20, the limitations of these claims substantially correspond to the limitations of claims 17 and 18, respectively; thus they are rejected on similar grounds as their corresponding claims.
Response to Arguments
On pages 9 and 10 of the Remarks, the Applicant's argument summarizes paragraphs 31 and 36 of Hold, and paragraphs 92 and 104 of Sytnik, and then asserts that “Hold and Sytnik fail to disclose or suggest acquiring attribute when image is shot and using it to perform effect processing.” The Examiner respectfully disagrees with this argument.
It is respectfully submitted that the cited art is replete with such disclosure. As just one example, the Remarks correctly quote paragraph 31 of Hold, which describes an “image editing system, performing a process 100 of extracting lighting conditions from an input digital image 102 and using the extracted lighting conditions to generate an output digital image 110 comprising an object rendered and illuminated according to the extracted lighting conditions.” Hold then continues with cited paragraph 32 (which is omitted in the Remarks), stating that “As used herein, ‘lighting conditions’ are the illumination properties of the environment in which a digital image was captured at the time it was captured.” The remaining paragraphs cited in the rejection, such as paragraph 38 (also omitted in the Remarks), among others, then elaborate on these attributes at time of capture and how they are used to perform effect processing to produce an output image. Thus, the Examiner respectfully submits that this clearly and explicitly teaches acquiring an attribute when an image is shot and using it to perform effect processing, that the prior art teaches these limitations.
On page 10 of the Applicant's Remarks, the Applicant argues that the remaining independent claims are not taught by the prior art for reasons similar to those discussed in regard to claim 1, and that the dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art. The Examiner respectfully disagrees with these arguments, for the reasons discussed above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID T WELCH whose telephone number is (571)270-5364. The examiner can normally be reached Monday-Thursday, 8:30-5:30 EST, and alternate Fridays, 9:00-2:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DAVID T. WELCH
Primary Examiner
Art Unit 2613
/DAVID T WELCH/Primary Examiner, Art Unit 2613
Prosecution Insights