Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Application has canceled claims 2, and non-elected claims 13-14, 16, and 18. As such, the application has pending claims 1,3-12,15 and 17.
Response to Arguments
Applicant’s arguments with respect to claims 1, 3, 5-8, 10 and 11, have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Thus, the new rejection of SONG, in combination with CHEN, LI, CAROLINE, Herley (cited in the previous office action), and newly cited prior art of Ivan, disclose the amended limitations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 10, 12, 15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Xu-bo SONG CN-108334878-A, hereinafter SONG, in further view of Bo-heng CHEN CN-110163188-A, hereinafter CHEN, Dian LI CN-104735465-B, hereinafter LI, BAILLARD CAROLINE EP-3336805-A1, hereinafter CAROLINE, Cormac E. Herley US-20020146173-A1, hereinafter Herley, and Zagaynov Ivan Germanovich RU-2631765-C1, hereinafter Ivan.
As per claim 1, SONG discloses a method of embedding an image in a video, comprising:inputting a video frame image of a video into a plane prediction model (see SONG page 5/26, wherein an input image is given to the SSD network, i.e., learning model capable of prediction, or prediction model), and acquiring a predicted plane region of the video frame image as well as 4 key points in the plane region (see SONG page 5/26, wherein the region of interest acquired in the input image is a plane. The detection layer is also acquired with the ROI. See further top of page 7/26, where this is done in step 3 after the training and at the neural network predicts the position of the plane. See also SONG page 5/26, wherein the boundary frame (e.g., 4 points of the frame) is disclosed), wherein the plane prediction model is obtained by training a deep learning model using training images with labels of a plane detection frame and a plane region as well as labeling information of the 4 key points in the plane region (see SONG page 5/26 step S12, wherein training the SSD neural network with the training data set, which comprises images as disclosed in step S11, and the target label information of the boundary frame, where the ROI is located, in the detection layer of the SDD network. See also SONG page 5/26 step S12, wherein the label information of the boundary frame, wherein the ROI is located, is disclosed).
While SONG does disclose a predicted plane region (see SONG page 7/26, wherein the neural network predicts the region of interest plane), SONG fails to explicitly disclose where CHEN discloses:a predicted plane mask of the video frame image (see CHEN bottom of page 7/52, wherein the foreground mask image is predicted from the non-reference frame picture in the video); andembedding the image to be embedded into the predicted plane mask of the video frame image (see CHEN bottom of page 16/52 and FIG. 15, wherein the target object is embedded into the foreground mask video).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s method by using CHEN’s teaching by including the image to be embedded to the predicted plane mask in order to improve the labeling information of the predicted plane.
However, SONG, in combination with CHEN, fails to explicitly dilosce where LI teaches: aligning 4 vertexes of the image to be embedded with the 4 key points in the predicted plane mask of the video frame image (see LI bottom of page 5/24, wherein the pixel points are set as the 4 vertex point of the advertisement plane area, or region. See further LI page 6-7/24, wherein the region adjustment process, i.e., aligning, involves the 4 points of the advertisement plane pattern region to be the implant plane), and embedding the image to be embedded into a position region corresponding to the 4 key points in the predicted plane mask of the video frame image (see LI page 7/24, wherein the pixel points are implanted into the pattern advertisement plane region), wherein the labeling information of the 4 key points in the plane mask in the training images is determined by a boundary line (see LI page 5/24 and FIG. 2, wherein the extending lines ba and cd, as well as da and cb is determined in the implant plane) and an inscribed rectangle of the plane mask in a plane coordinate system (see LI page 5/24 and FIG. 2, wherein the wall area abcd is determined in the implant plane to create the inscribed rectangle).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, method by using LI’s teaching by including the alignment of vertexes to the image to be embedded in order to improve the orientation of the image on the predicted plane mask.
However, SONG, in combination with CHEN and LI, fails to explicitly disclose where CAROLINE teaches:performing Hough line detection on an edge of the plane mask in the plane coordinate system (see CAROLINE page 5/29 Step 11, wherein Hough transform, a line extraction technique, is disclosed using the 3D point cloud);determining a probability that each detected line is the boundary line of the plane mask (see CAROLINE page 5/29 Step 11, wherein a threshold, or probability, on edge length is disclosed).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN and LI, method by using CAROLINE’s teaching by including edge and line detection to the boundary lines of the plane mask in order to accurately acquire the limitations of the plane mask.
However, SONG, in combination with CHEN, LI, and CAROLINE, fails to explicitly disclose where Herley teaches:choosing a pair of lines having a perpendicular relation or a parallel relation from the detected lines (see Herley ¶21-23 and FIG. 2, wherein the detected line edges are determined to have a parallel relation with each other).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, and CAROLINE, method by using Herley’s teaching by choosing the pair of lines with a parallel or perpendicular relation in the detected lines in order to further determine which lines would make the boundary of the plane mask.
However, SONG, in combination with CHEN, LI, CAROLINE, and Herley, fails to explicitly disclose where Ivan teaches:determining a line with the highest probability in the pair of lines or the detected lines as one boundary line of the plane mask in the plane coordinate system, depending on whether the pair of lines is found (see Ivan ¶37, wherein the battery array is used to select a plurality of lines to be the binding line. After processing all the pixels, a search is made in the battery array of which are the most probable lines in the image. The binding line, as disclosed in ¶35, is the border line, wherein the fold, or cut line, of an image is determined to be).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, and Herley, method by using Ivan’s teaching by including a determination of the highest probability to the pair of lines in order to further confirm that the lines make up the boundary of the plane mask.
As per claim 3, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, wherein the labelling information of the 4 key points in the plane region in each training image is obtained (see SONG page 5/26 step S12, wherein the label information of the boundary frame is disclosed, wherein the default boundary frame (e.g., 4 points of the frame) of the target object is also calculated) by: converting the plane region of the each training image from a pixel coordinate system to a plane coordinate system (see LI page 5-6/24 step 103, wherein pixel point area, or region, is transformed to be implanted into the coordinates of the area, or region, of the plane pattern advertisement); determining a boundary line of the plane region in the plane coordinate system (see LI page 5/24 and FIG. 2, wherein the extending lines ba and cd, as well as da and cb is determined in the implant plane); determining an inscribed rectangle of the plane mask in the plane coordinate system based on the boundary line of the plane mask (see LI page 5/24 and FIG. 2, wherein the wall area abcd is determined in the implant plane to create the inscribed rectangle); and converting 4 vertices of the inscribed rectangle of the plane region from the plane coordinate system to the pixel coordinate system (see LI bottom of page 7/24, wherein after the coordinates of the corresponding 4 pixel points are determined, the changed pixel value is obtained).
As per claim 5, while SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, where performing Hough line detection on an edge of the plane mask in the plane coordinate system (see CAROLINE page 5/29 Step 11, wherein Hough transform is used on the candidate area where the edges are detected) comprises: performing edge detection on the plane mask in the plane coordinate system (see CAROLINE page 5/29 Step 11 and FIG. 3, wherein candidate areas are identified in the 3D-scene, i.e., plane coordinate area, or region. The candidate areas are delineated, i.e., edges are detected);and performing Hough line detection on the plane mask in the plane coordinate system based on a detected edge of the plane mask (see CAROLINE page 5/29 Step 11, wherein Hough transform, a line extraction technique, is disclosed using the 3D point cloud).
As per claim 10, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, wherein the embedding the image to be embedded into the predicted plane region of the video frame image (see LI page 7/24, wherein the pixel points are implanted into the pattern advertisement plane region) comprises: determining a transformation matrix from the image to be embedded to the predicted plane region of the video frame image according to a mapping relation between the 4 vertexes of the image to be embedded and the 4 key points in the plane region of the predicted video frame image (see LI page 6/24, wherein the transformation matrix is corresponding between the pixel point area and the advertisement image is disclosed. see further page 7/24, wherein the 4 coordinate points between the images are disclosed); and transforming each foreground point of the image to be embedded into the position region corresponding to the 4 key points in the predicted plane region of the video frame image based on the transformation matrix (see LI page 7-8/24, wherein the points on the target image are implanted, or embedded, onto the plane pattern advertisement region using the coordinate of 4 set of pixel points).
As per claim 12, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, wherein the deep learning model (see SONG page 5/26 step S12, wherein the SSD network uses a convolutional neural network) comprises region-based convolutional neural networks (see CHEN page 10/52, wherein deep learning with R-CNNs, regional convolutional neural networks, is disclosed); or the image to be embedded comprises an enterprise identification image and a product image.
As per claim 15, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses an apparatus of embedding an image in a video, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to perform the method of embedding an image in a video according to claim 1 based on instructions stored in the memory (see CHEN page 4/26, wherein a memory and processor to execute instructions is disclosed).
As per claim 17, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses a non-transitory computer-readable storage medium comprising a stored computer program which implements the method of embedding an image in a video according to claim 1 when executed by a processor (see CHEN top of page 4/26, wherein a computer-readable medium is disclosed).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, in further view of Chen Liu et al., PlaneRCNN, hereinafter Liu.
As per claim 4, while SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses wherein the converting the plane mask of the each training image from a pixel coordinate system to a plane coordinate system (see LI page 5-6/24 step 103, wherein pixel point area, or region, is transformed to be implanted into the coordinates of the area, or region, of the plane pattern advertisement), it fails to explicitly disclose where Liu teaches comprising:converting the plane mask of the each training image from the pixel coordinate system to a world coordinate system (see Liu page 3-4/16, wherein the pixel coordinates are used for depth estimation to create the depth maps using the sampled training images in the Mask-RCNN. The depth maps of the segmentation plane masks are then used to create 3D coordinate maps); andconverting the plane mask of the each training image from the world coordinate system to the plane coordinate system (see Liu page 4/16 Section 3.3, wherein the 3D coordinate
p
c
is transformed to the coordinate frame, wherein the frame contains the planar region. This warping is done for the training samples, i.e., training images).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, method by using Liu’s teaching by including a world coordinate system to the pixel coordinate system of the plane mask in order to further acquire a real-world measurable scale of the image.
Claims 6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, in further view of MARINO WILLIAM L WO-2017165538-A1, hereinafter WILLIAM.
As per claim 6, while SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, wherein the determining a probability that each detected line is the boundary line of the plane region (see CAROLINE page 5/29 Step 11 and FIG. 3, wherein candidate areas are identified in the 3D-scene, i.e., plane coordinate area, or region, as well as a threshold, or probability, on edge length is disclosed), it fails to explicitly disclose where WILLIAM teaches comprising:determining the probability that each detected line is the boundary line of the plane region according to difference information of symmetrical regions on both sides of the each detected line (see WILLIAM ¶224-226, wherein the host region is identified using parallelogram lines, i.e., symmetrical lines, based on some predetermined threshold. The host region is the coordinates of corners of the bounding box in that frame, i.e., boundary line. See also ¶174, wherein the host region is identified to be a plane), wherein the smaller the difference between the symmetrical regions on the both sides of a line, the greater the probability that the line is the boundary line of the plane region (see WILLIAM ¶227-228 and FIG. 20, wherein subsequent frames are subtracted from each other, and if the difference is zero or substantially close, that part of the frame is determined to be the host region).
While WILLIAM discloses the smaller difference, it would have been obvious to one of ordinary skill to use a larger difference to measure the symmetrical regions. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, method by using WILLIAM’s teaching by including probability determination to the boundary line detection in order to further confirm the coordinates of the planar region.
As per claim 9, while SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 3, wherein the determining an inscribed rectangle of the plane region in the plane coordinate system (see LI page 5/24 and FIG. 2, wherein the wall area abcd is determined in the implant plane to create the inscribed rectangle using the plane pattern area coordinate as further disclosed in page 7/24), it fails to explicitly disclose where WILLIAM teaches comprising:determining an inscribed quadrilateral of the plane mask in the plane coordinate system, wherein the inscribed quadrilateral is parallel to the boundary line and comprises a maximum inscribed rectangle (see WILLIAM ¶186-187 and FIG. 14, wherein the maximally sized rectangle in the coordinates of the host region, which is a quadrilateral shape as disclosed in ¶232, is disclosed. The host region is parallelized with the screen as stated in ¶170. The host region is acquired in the plane of the video scene, as well as the coordinates of the plane, as disclosed in ¶98).
While WILLIAM does not explicitly disclose an inscribed rectangle and a maximally inscribed square, it would have been obvious to one of ordinary skill to use WILLIAM’s inscribed quadrilateral and maximally inscribed rectangle as an inscribed rectangle and a maximally inscribed square. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, method by using WILLIAM’s teaching by including an inscribed rectangle to the plane mask in order to further define a region within the plane mask.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, in further view of Jin-peng YANG CN-117647194-A, hereinafter YANG.
As per claim 8, while SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, discloses the method according to claim 1, wherein the determining a boundary line of the plane region in the plane coordinate system (see CAROLINE page 5/29 Step 11 and FIG. 3, wherein candidate areas are identified in the 3D-scene, i.e., plane coordinate area, or region, as well as a threshold, or probability, on edge length is disclosed), it fails to explicitly disclose where YANG further teaches further comprising:performing median filtering on the plane area, or region, in the plane coordinate system before the edge detection (see YANG page 18/57, wherein median filtering is performed before edge detection on the plane image); or merging the detected lines based on a slope of each line after the Hough line detection.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, method by using YANG’s teaching by including median filtering to the plane mask in order to clean up the plane mask for better edge detection results.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, in further view of LEMOINE GUILLAUME WO-2013087935-A1, hereinafter GUILLAUME and CHUEN HORNG LIN TW-201445509-A, hereinafter LIN.
As per claim 11, SONG, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, fails to explicitly disclose where GUILLAUME teaches:The method according to claim 1, wherein the deep learning model uses a loss function that is determined based on the 4 key points in the labelling information and the predicted 4 key points after the alignment operation is performed (see GUILLAUME bottom of page 4/29, wherein a deformation function, i.e., modified loss function, is used based on the predefined area of incrustation done by learning), wherein performing the alignment operation on the predicted 4 key points comprises: determining a transformation ratio based on the 4 key points in the labelling information and the predicted 4 key points (see GUILLAUME bottom of page 4/29, wherein the deformation of the incrustation zone is identified and the frame is deformed in the same way. The image to be embedded is trapezoidal, and the key points are disclosed on page 5/29); performing size transformation on the predicted 4 key points according to the transformation ratio (see GUILLAUME page 5/29, wherein the trapezoidal area is transformed using the transformation matrix and four extreme points); determining first position transformation information based on the 4 key points in the labelling information (see GUILLAUME page 5/29, wherein the incrustation area corresponds to the 4 points of the trapezoid to be embedded); determining second position transformation information based on the predicted 4 key points (see GUILLAUME page 5/29, wherein the keying area corresponds to the 4 points of the trapezoid to be embedded, as well as its shape and position, i.e., transformation information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, and Ivan, method by using GUILLAUME’s teaching by including an alignment and transformation operation to the deep learning model in order to more accurately align and transform the labeling information of the plane mask.
However, SONG, in combination with CHEN, LI, CAROLINE, Herley, Ivan, and GUILLAUME, fails to explicitly disclose where LIN teaches:respectively adding the first position transformation information to the predicted 4 key points after the size transformation and subtracting the second position transformation information, to finish the alignment operation on the predicted 4 key points (see LIN page 5/33, wherein a wavelet transformation, which can measure location information and transformation, is used to then add and subtract values to obtain the image using the reference alignment).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify SONG’s, in combination with CHEN, LI, CAROLINE, Herley, Ivan, and GUILLAUME, method by using LIN’s teaching by modifying the transformation information to add the first position transformation information and subtract the second position transformation in order to improve the results of the finished alignment operation.
Allowable Subject Matter
Claim 7 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bradley Obas Felix whose telephone number is (703)756-1314. The examiner can normally be reached M-F 8-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 5712728243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRADLEY O FELIX/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights