DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: The punctuation error on line 193 of the specification “a generating module configured to, in response to that the target video frame satisfies a preset drawing condition, generate a drawn image associated with the target object based on contour information of the target object in the target video frame,;”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-13, & 20 are rejected under 35 U.S.C. 103 as being unpatentable over Krissy’s Krazy Fun Channel (“Chuck E Cheese Sketch Book Arcade Drawing Game ~ Draw Sketch Your Picture Game”) (herein after “Krissy”) in view of Jia-Bin Huang (U.S. Pat. App. Pub. No. CN 110288519 A, herein after “Huang”).
Regarding claims 1, 13 & 20, Krissy discloses a method for processing media content (“Chuck E Cheese Sketch Book Arcade Drawing Game ~ Draw Sketch Your Picture Game), comprising: displaying video media content in a display interface (Timestamp :01-1:00),
the video media content comprising at least one target video (Timestamp :24)
PNG
media_image1.png
1068
1472
media_image1.png
Greyscale
frame with a target object (Timestamp :15);
PNG
media_image2.png
1043
1466
media_image2.png
Greyscale
The frame at timestamp :15 seconds shows a “Chuck E Cheese” arcade machine displaying an image with a target object, a child with a doll. Further, a later frame at timestamp :23 seconds shows that the image is a frame in a video of the baby with the doll.
generating a drawn image associated with the target object… (Timestamp 1:01)
PNG
media_image3.png
1065
1465
media_image3.png
Greyscale
and switching to display, in the display interface, a preset virtual brush and a drawing process of the drawn image by the virtual brush (Timestamp :25-:34).
PNG
media_image4.png
1070
1463
media_image4.png
Greyscale
The drawn image appears gradually starting at timestamp :29 seconds in Krissy’s uploaded video, and the complete product can be viewed on screen and printed as seen in timestamp 1:00 minute.
Krissy fails to expressly disclose that in response to that the target video frame satisfies a preset drawing condition, and generation of the drawn image associated with the target object is based on contour information of the target object in the target video frame.
Huang teaches in response to that the target video frame satisfies a preset drawing condition (“calculating the output probability of the combined to the edge critical point between the skeleton key set, the candidate key point edge key point probability of the output is 1 of the key candidate edge point is determined to be the skeleton key point matching, the output probability is 0 determined as the edge key point of the skeleton key point does not match”; Huang, page 3) where output probability is the odds of a target image being displayed if the frame corresponds to required preset key points.
Huang additionally teaches generation of the drawn image associated with the target object is based on contour information of the target object in the target video frame (“… the body contour region to perform deformation operation to obtain the human body outline deformation image”; Huang, page 10.) where a contour region is found and modified to produce an output deformation image, and (“… the key point set based on the edge, determining the body contour of the target object, comprising: obtaining the edge key point of each partial matching frame after the key point set, using the connecting method of the Hermite interpolation, connected to the respective edge of the key point, generating an object body contour.”; Huang, page 3).
In regards to claim 13, claim 1 is substantially similar to claim 13 hence the rejection analysis for claim 1 is also applied to claim 13. Krissy also teaches the additional limitations of claim 13 which recites an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising (Timestamp between :00-1:00).
In regards to claim 20, claim 1 is substantially similar to claim 20 hence the rejection analysis for claim 1 is also applied to claim 20. Krissy also teaches the additional limitations of claim 20 which recites a non-transitory computer readable storage medium storing computer execution instructions, the computer execution instructions, when executed by a processor, implementing acts comprising… the video from Krissy between :00-1:00 teaches an arcade machine, which includes a non-transitory computer readable storage medium storing computer execution instructions.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of obtaining a target video frame with target object, and display interface with virtual brush for drawing process of the target object taught by Krissy, with the teachings of Huang to include a method of output image generation based on contour information, provided by the target video frame.
The suggestion/motivation to do so would have been to provide user entertainment, image enhancement, and to allow for beautification methods to be applied as noted by Krissy and Huang. Implementing the methods taught by Huang to determine key points of a target object to establish a contour line, with Krissy to display an image and draw the target image would yield the predictable result of a drawn target image based on contour information.
Regarding claims 2 & 14, Krissy in view of Huang teaches that in response to that the target video frame satisfies the preset drawing condition, generating the drawn image associated with the target object based on the contour information of the target object in the target video frame comprises: (Krissy, timestamp :01-1:00 & Huang, page 10)
performing a recognition operation on the target object in the target video frame to obtain a recognition result(“based on the human body key point, determining the target object of the first centre point and the second centre point; based on the first centre point and the second centre point, determining the target object body centre line.”; Huang, page 2, where the method of determining a target object is known to be a recognition operation that finds a target object based on key points, where “the key point set based on the edge, determining the body contour of the target object.” (Huang, page 3));
determining whether the target object satisfies the preset drawing condition based on the recognition result (“calculating the output probability of the combined to the edge critical point between the skeleton key set, the candidate key point edge key point probability of the output is 1 of the key candidate edge point is determined to be the skeleton key point matching, the output probability is 0 determined as the edge key point of the skeleton key point does not match”; Huang, page 3); and
in response to that the target object satisfies the preset drawing condition, generating the drawn image associated with the target object based on the contour information of the target object(“… According to a particular implementation of the present disclosure, the key point set based on the edge, determining the body contour of the target object, comprising: obtaining the edge key point of each partial matching frame after the key point set, using the connecting method of the Hermite interpolation, connected to the respective edge of the key point, generating an object body contour.”; Huang, page 3).
In regards to claim 14, claim 2 is substantially similar to claim 14 hence the rejection analysis for claim 2 is also applied to claim 14. Krissy also teaches the additional limitations of claim 14 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the methods of a recognition operation used for a recognition result that checks for compliance with a preset drawing condition taught by Huang, with the teachings of Alvarez to generate the image based on contour information of the target object.
The suggestion/motivation to do so would have been to make a photo more refined and provide emphasis on a specific aspect of a photo, where “the embodiment of the present disclosure provides an image beautification method … on the target image of the human body key point, based on the human body key point, determining body contour and head protection region.” (Huang, page 3).
Regarding claims 6 & 18, Krissy in view of Huang teaches of switching to display, in the display interface, the preset virtual brush and the drawing process of the drawn image by the virtual brush comprises: (Krissy, timestamp :01-1:00)
displaying, in a first display area of the display interface, the preset virtual brush and the drawing process of the drawn image by the virtual brush (Krissy, timestamp :01-1:00) it can be seen in the video that a girl with a doll are pictured in a first display, and after their photo is taken, a virtual brush draws them both; and
displaying, in a second display area of the display interface, the target object in the video media content (Krissy, timestamp :00) there is a second display on the lower portion arcade machine for outputting instructions and the live video, including the target object.
PNG
media_image5.png
953
1321
media_image5.png
Greyscale
In regards to claim 18, claim 6 is substantially similar to claim 18 hence the rejection analysis for claim 6 is also applied to claim 18. Krissy also teaches the additional limitations of claim 18 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
Regarding claims 7 & 19, Krissy in view of Huang teaches the method of claim 1,
wherein the method further comprises: after switching to display, in the display interface, the preset virtual brush and the drawing process of the drawn image by the virtual brush, (Krissy, timestamp :01-1:00 & Huang, page 6)
determining a decoration masking to be processed (“the target object in the target image is executing key point detection so as to obtain the target object on the human body key point of the target image. the target image comprises the image to be taking beautifying processing, … target image can be shot by the intelligent mobile phone and other device also can be the image obtained by other ways.” (Huang, page 6)) where beautification can be applied manually or automatically to the body based on positioning determined by a detection module, i.e. decoration masking. Additionally, Krissy at timestamp :29 seconds teaches a virtual brush which is used to display the target image with target object.;
performing an adjustment operation on the decoration masking to be processed based on the drawing animation to obtain a target decoration masking (“adjusting the target object to display on the target image to obtain the effect of a real beautification.” (Huang, page 6)) where the target object being adjusted includes the key points used for beautification, therefore changing the target decoration masking. Additionally, Krissy teaches a drawing animation at timestamp :29 seconds, which can be applied to a decorated image; and
dynamically displaying, on the drawn image, a drawing process of the target decoration masking by the virtual brush (Krissy, timestamp :29) Krissy teaches a dynamic drawing animation, with a virtual brush. Additionally, Huang teaches a method of key point detection on an image which can be used to apply beautification methods (Huang, page 6).
In regards to claim 19, claim 7 is substantially similar to claim 19 hence the rejection analysis for claim 7 is also applied to claim 19. Krissy also teaches the additional limitations of claim 19 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of a display interface containing a preset virtual brush taught by Krissy, with the teachings of Huang to include a method of dynamic decoration via the use of masking.
The suggestion/motivation to do so would have been to provide user entertainment, image enhancement, and to allow for beautification methods to be applied as noted by Krissy and Huang. Implementing the methods taught by Huang to determine key points of a target object with Krissy to display an image and use an animation to draw the target image would yield the predictable result of a drawn target image which can be dynamically decorated, based on the masking.
Regarding claim 8, Krissy in view of Huang teaches the method of claim 7,
wherein determining the decoration masking to be processed comprises: (Huang, page 5)
in response to a decoration operation triggered by a user, displaying at least one preset decoration masking (“image beautification method provided by this embodiment is executed by a computing device, the computing device may be implemented as software, or as a combination of software and hardware, the computing device can be integrated in the server, terminal device and so on.” (Huang, page 5)) where Huang does not limit the computing device and software to operate from being triggered by a human interaction, or to operate automatically.; and
in response to a selection operation of a user on the at least one decoration masking, determining a decoration masking selected by the user as the decoration masking to be processed (“the memory is stored with instructions … so that the at least one processor capable of executing image beautification method in the any of the first aspect or any one of the first aspect of an implementation.” (Huang, page 5)) where Huang does not limit the computing device and software to operate from being triggered by a human interaction, or to operate automatically, a selected masking stored in memory would be executed by the processor.
Regarding claim 9, Krissy in view of Huang teach the method of claim 7,
wherein performing the adjustment operation on the decoration masking to be processed based on the drawing animation to obtain the target decoration masking comprises: (Huang, page 6)
determining position information of at least one key point corresponding to the target object (“after obtaining the human body centre line, … based on the body center line of body contour for deformation operation. specifically, the body center line the body contour is divided into left and right two parts, can be compression deformation towards the centre line direction through the left and right parts.” (Huang, page 7)); and
performing a size scaling operation and/or a shape adjustment operation on the decoration masking to be processed based on the position information of the at least one key point, to obtain the target decoration masking (“based on the human body key point, determining the human body outline and head protection region … performed on the human body outline area deformation operation to obtain the human body outline deformation image.” (Huang, page 4)) where the key points are used to determine the body outline for accurate target image deformation/beautification.
PNG
media_image6.png
378
348
media_image6.png
Greyscale
Regarding claim 10, Krissy in view of Huang teach the method of claim 9,
wherein dynamically displaying on the drawn image the drawing process of the target decoration masking by the virtual brush comprises: (Krissy, timestamp :01-1:00 & Huang, page 6)
determining a display position of the target decoration masking on the drawn image according to the position information of the at least one key point (“based on the first centre point and the second centre point, determining the target object body centre line” (Huang, page 7)) where key points are used to determine the position and location of the target object within the target image, and the beautification methods are applied within the contour of the key points; and
dynamically displaying the drawing process of the target decoration masking by the virtual brush on the drawn image based on the display position (Krissy, timestamp :29) Krissy teaches a method of dynamically displaying a drawing process of an image by use of a virtual brush.
Regarding claim 11, Krissy in view of Huang teach the method of claim 7,
wherein the decoration masking comprises a human face processing masking and/or a human body processing masking (“the target object in the target image is executing key point detection so as to obtain the target object on the human body key point of the target image.” (Huang, page 5)) it is noted that using key points to detect portions of the human body also apply to the head and face, beautification masking can be done through these key points and their locations, therefore, Huang teaches beautification masking for the body and/or face.
Regarding claim 12, Krissy in view of Huang teach the method of claim 7,
further comprising:
in response to a generation operation triggered by a user, generating target media content based on the drawing process corresponding to the drawn image and/or the drawing process of the target decoration masking (“communication device 509 may allow electronic device 50 with other device for wireless or wired communication to exchange data.” (Huang, page 11)) Huang notes an I/O interface may allow communication of data such as uploading or downloading. Additionally, Krissy at timestamp 1:00 minute shows the output of the drawing containing a print feature, and at timestamp :01 seconds, a generation operation must be selected by the user.
Claim(s) 3-5 & 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Krissy, Huang, and further in view of Tong Li et. al. (U.S. Pat. App. Pub. No. US 20210035260 A1, herein after “Tong”).
Regarding claims 3 & 15, Krissy in view of Huang teaches the method of claim 2,
in response to determining, according to the recognition result, that an expression feature of the target object matches a preset target expression and/or an action feature of the target object matches a preset target action, determining that the target object satisfies the preset drawing condition(“After the target image formed, can aim at the head area and human body area key point detection, so as to obtain the plurality of head region key point and a plurality of human body area key point. key point detection of the head area and human body area can adopt such as CPM (Convolutional Pose Machine, convolution gesture detector)”; Huang, page 6) where it is known that a convolutional pose machine (CPM) is a sequence of networks that map a location of identified parts such as fingers on a hand, or ligaments on a body.
Krissy and Huang do not teach wherein determining whether the target object satisfies the preset drawing condition based on the recognition result comprises at least one of:
in response to determining, according to the recognition result, that the target object comprises a preset target key point, determining that the target object satisfies the preset drawing condition.
Tong teaches wherein determining whether the target object satisfies the preset drawing condition based on the recognition result comprises at least one of:
in response to determining, according to the recognition result, that the target object comprises a preset target key point, determining that the target object satisfies the preset drawing condition (“… a detection unit and a deformation processing unit, where the detection unit is configured to obtain a first image, identify a target object in the first image, and obtain leg detection information of the target object”; Tong, ¶ [0007]) where the key point information is based on a target line which may also be referred to as a center line, and the resulting contour lines may be used to modify the image.
In regards to claim 15, claim 3 is substantially similar to claim 15 hence the rejection analysis for claim 3 is also applied to claim 15. Krissy also teaches the additional limitations of claim 15 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the methods of target points observed on a target object, used to determine a preset condition taught by Tong, with the teachings of Huang to produce a recognition function that can observe and identify target expressions or actions.
The suggestion/motivation to do so would have been to allow for more accurate, and on-demand image processing. “…to implement automatic adjustment … of the target object without multiple manual operations of a user, which greatly improves operational experiences of the user.” (Tong, ¶ [0072]).
Regarding claims 4 & 16, Krissy in view of Huang teaches the method of claim 3,
performing a drawing operation on the contour information by a preset target brush, to obtain a first drawing result (Huang, page 3 and Krissy, Timestamp :29-1:01) where in this portion of the video, a virtual brush is drawing the target object to produce a drawing result.
PNG
media_image4.png
1070
1463
media_image4.png
Greyscale
Krissy and Huang do not teach wherein generating the drawn image associated with the target object based on the contour information of the target object comprises:
recognizing contour information corresponding to the target object, wherein the contour information comprises outer contour information and inner contour information;
determining a hair region contour corresponding to the target object by using a preset hair segmentation algorithm, and
obtaining a second drawing result based on the hair region contour
Tong teaches wherein generating the drawn image associated with the target object based on the contour information of the target object comprises:
recognizing contour information corresponding to the target object, wherein the contour information comprises outer contour information and inner contour information (“It may be understood that the leg region of the target object has an inner contour and an outer contour, that is, the leg region may have an outer contour line and an inner contour line.“ (Tong, ¶ [0025]));
determining a hair region contour corresponding to the target object by using a preset hair segmentation algorithm, and obtaining a second drawing result based on the hair region contour (“the plurality of sub-regions include a first sub-region and a second sub-region. The first sub-region and the second sub-region correspond to a thigh region of the leg region.” (Tong, ̬¶ [0043])) where the preset sub-regions can be designated to an adjacent part of the human body;
obtaining the drawn image associated with the target object based on the first drawing result and the second drawing result (“… the plurality of sub-regions include a first sub-region and a second sub-region. The first sub-region and the second sub-region correspond to a thigh region of the leg region.” (Tong, ¶ [0082])) where there may be multiple sub-regions of adjacent body parts, and the final image is obtained through the combination of multiple sub-regions.
In regards to claim 16, claim 4 is substantially similar to claim 16 hence the rejection analysis for claim 4 is also applied to claim 16. Krissy also teaches the additional limitations of claim 16 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the methods of drawing a target object with contour information by a target brush by Krissy and Huang with the teachings of Tong to perform a second contour recognition, and combine the two to produce the drawn image.
The suggestion/motivation to do so would have been to identify multiple key points for later modifications. “… performing compression processing … or performing stretching processing … according to a direction from the target line to the contour line.” (Tong, ¶ [0027]).
Regarding claims 5 & 17, Krissy in view of Huang and Tong teach the method of claim 4,
wherein determining the hair region contour corresponding to the target object by using the preset hair segmentation algorithm comprises: (Tong, ¶ [0043])
performing a segmentation operation on the target object by the preset hair segmentation algorithm, to obtain a target mask corresponding to the hair region (“outline the region … of the key point comprises, eyes, nose and ear organs such as the key point set,” (Huang, page 9)) where the outline has an output probability based on the edge critical point, the output probability determines what portion of the human body will be displayed, therefore segmenting it for modifications, (“After determining a graphic for the head region, according to the head area enlarging coefficient, in a head area pattern is central, outwards expanding pattern area, obtaining the head protection area.” (Huang, page 9)) where a head protection area is separate from the human body outline, it is understood that the values to determine a protection area could be altered for another part of the human body, such as the hair.
Determining the second drawing result corresponding to the target object according to a preset channel value range and the target mask (“… the candidate key point edge key point probability of the output is 1 of the key candidate edge point is determined to be the skeleton key point matching, the output probability is 0 determined as the edge key point of the skeleton key point does not match” (Huang, page 3)) where 0 to 1 is a preset channel value range, the candidate edge is tested to match the skeleton key set, and the process can be repeated for multiple portions of the human body for a plurality of image results.
In regards to claim 17, claim 5 is substantially similar to claim 17 hence the rejection analysis for claim 5 is also applied to claim 17. Krissy also teaches the additional limitations of claim 17 which recites the device of claim 13… the video from Krissy between :00-1:00 teaches an arcade machine, which includes an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory, to cause the processor to perform acts comprising.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611