DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7, 17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU).
Regarding claim 1.
Long teaches:
A method comprising:
generating, by at least one processing device utilizing a segmentation neural network, a segmentation mask of a planar surface in a digital image (Long [0030] Image segmentation component 130 may identify image segmentation information for the 2D image. Image segmentation component 130 may also extract object information using a mask regional convolutional neural network (Mask R-CNN). The image segmentation information is based on the object information.);
Long fails to teach:
determining, by the at least one processing device utilizing a corner detection model, coordinates of corners of the planar surface from an outer contour extracted from the segmentation mask (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
generating, by the at least one processing device and for display via a graphical user interface displaying the digital image, a modified digital image comprising a digital asset inserted onto the planar surface by performing a perspective transformation on the digital asset utilizing a transformation matrix determined from the coordinates of the corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
“LYFE Marketing” teaches:
determining, by the at least one processing device utilizing a corner detection model, coordinates of corners of the planar surface from an outer contour extracted from the segmentation mask (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
generating, by the at least one processing device and for display via a graphical user interface displaying the digital image, a modified digital image comprising a digital asset inserted onto the planar surface by performing a perspective transformation on the digital asset utilizing a transformation matrix determined from the coordinates of the corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with “LYFE Marketing”. Being able to change different places in an image with a different image using artificial ai, as in “LYFE Marketing”, would benefit the Long teachings by allowing for a way to create custom product inserts or billboard inserts. Additionally, this is the application of a known technique, using a model to be able to insert different objects or pictures into an image, to yield predictable results.
Regarding claim 7.
Long and “LYFE Marketing” teach:
The method of claim 1, further comprising:
determining a training dataset comprising a plurality of filtered digital images with a specified set of visual attributes (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.);
generating a modified training dataset by filtering the training dataset to a subset of digital images of the plurality of filtered digital images classified as comprising planar surfaces by a classifier model (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.); and
training the segmentation neural network using the training dataset to generate segmentation masks for planar surfaces in digital images (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with “LYFE Marketing”. Being able to change different places in an image with a different image using artificial ai, as in “LYFE Marketing”, would benefit the Long teachings by allowing for a way to create custom product inserts or billboard inserts. Additionally, this is the application of a known technique, using a model to be able to insert different objects or pictures into an image, to yield predictable results.
Regarding claim 17.
Long teaches:
A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising: generating, utilizing a segmentation neural network, a segmentation mask for a planar surface in a digital image (Long [0030] Image segmentation component 130 may identify image segmentation information for the 2D image. Image segmentation component 130 may also extract object information using a mask regional convolutional neural network (Mask R-CNN). The image segmentation information is based on the object information.);
Long fails to teach:
determining, utilizing a corner detection model, coordinates of corners of the planar surface from an outer contour extracted from the segmentation mask (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
generating, for display via a graphical user interface displaying the digital image, a modified digital image comprising a digital asset inserted onto the planar surface by performing a perspective transformation on the digital asset utilizing a transformation matrix determined from the coordinates of the corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
“LYFE Marketing” teaches:
determining, utilizing a corner detection model, coordinates of corners of the planar surface from an outer contour extracted from the segmentation mask (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
generating, for display via a graphical user interface displaying the digital image, a modified digital image comprising a digital asset inserted onto the planar surface by performing a perspective transformation on the digital asset utilizing a transformation matrix determined from the coordinates of the corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with “LYFE Marketing”. Being able to change different places in an image with a different image using artificial ai, as in “LYFE Marketing”, would benefit the Long teachings by allowing for a way to create custom product inserts or billboard inserts. Additionally, this is the application of a known technique, using a model to be able to insert different objects or pictures into an image, to yield predictable results.
Regarding claim 19.
Long and “LYFE Marketing” teach:
The non-transitory computer readable medium of claim 17, wherein generating the modified digital image comprises:
generating a transformation matrix from source points corresponding to corners of the digital asset to destination points corresponding to the coordinates of the corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
executing a function to perform the perspective transformation on the digital asset by applying the transformation matrix to a plurality of pixels of the digital asset (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with “LYFE Marketing”. Being able to change different places in an image with a different image using artificial ai, as in “LYFE Marketing”, would benefit the Long teachings by allowing for a way to create custom product inserts or billboard inserts. Additionally, this is the application of a known technique, using a model to be able to insert different objects or pictures into an image, to yield predictable results.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU) and Zhang et al. (CN 110717852).
Regarding claim 2.
Long and “LYFE Marketing” teach:
The method of claim 1,
Long and “LYFE Marketing” fail to teach:
wherein determining the coordinates of the corners of the planar surface comprises: generating a contour image by extracting the outer contour from the segmentation mask; and filling, in the contour image, an interior area defined by the outer contour with a solid fill color (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,).
Zhang teaches:
wherein determining the coordinates of the corners of the planar surface comprises: generating a contour image by extracting the outer contour from the segmentation mask; and filling, in the contour image, an interior area defined by the outer contour with a solid fill color (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and “LYFE Marketing”. Obtaining a contour image, as in Zhang, would benefit the Long and “LYFE Marketing” teachings by using specific operators to obtain the contour of an image to be able to segment. Additionally, this is the application of a known technique, obtaining a contour image, to yield predictable results.
Claim(s) 3, 4, 5, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU), Zhang et al. (CN 110717852), and Huo et al (CN 117037083).
Regarding claim 3.
Long, “LYFE Marketing”, and Zhang teach:
The method of claim 2, wherein determining the coordinates of the corners of the planar surface comprises:
generating an edge map by detecting a plurality of edges in the contour image utilizing an edge detection algorithm (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,); and
Long, “LYFE Marketing”, and Zhang fail to teach:
determining, from the edge map, sets of one or more straight lines along the outer contour from the plurality of edges by utilizing a Hough transform to (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.):
generate representations of the plurality of edges in a polar coordinate system utilizing pairs of polar coordinates (Huo [Pg 8 Par 3] Specifically, firstly, establishing a two-dimensional plane coordinate system in the first image, for each lane solid line edge point, obtaining a plurality of the first coordinate and the second coordinate; then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form,); and
determine the sets of one or more straight lines along the outer contour by counting a number of edge points of the plurality of edges that fall on each pair of polar coordinates and iterating over a plurality of threshold intersection values (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space. for each lane solid line edge point, adding one operation at the corresponding position in the accumulator according to the parameter value;).
Huo teaches:
determining, from the edge map, sets of one or more straight lines along the outer contour from the plurality of edges by utilizing a Hough transform to (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.):
generate representations of the plurality of edges in a polar coordinate system utilizing pairs of polar coordinates (Huo [Pg 8 Par 3] Specifically, firstly, establishing a two-dimensional plane coordinate system in the first image, for each lane solid line edge point, obtaining a plurality of the first coordinate and the second coordinate; then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form,); and
determine the sets of one or more straight lines along the outer contour by counting a number of edge points of the plurality of edges that fall on each pair of polar coordinates and iterating over a plurality of threshold intersection values (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space. for each lane solid line edge point, adding one operation at the corresponding position in the accumulator according to the parameter value;).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Regarding claim 4.
Long, “LYFE Marketing”, Zhang, and Huo teach:
The method of claim 3, wherein determining the coordinates of the corners of the planar surface comprises:
filtering the sets of one or more straight lines by comparing distances between pairs of lines of the sets of one or more straight lines to a distance threshold (Huo [Pg 8 Par 3] then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form, that is using distance p and angle theta to represent the straight line, a plurality of said parameters correspond to a plurality of straight lines passing through the edge point of said lane solid line;); and
filtering the sets of one or more straight lines by comparing angles between the pairs of lines of the sets of one or more straight lines to an angle threshold (Huo [Pg 8 Par 3] then, the intersection analysis according to the first straight line parameter and the second straight line parameter is to convert a plurality of straight line families passing through a plurality of lane solid line edge points into a plurality of lines in a parameter space according to a plurality of the parameters, wherein the straight line family corresponds to the line in the parameter space one by one;).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Regarding claim 5.
Long, “LYFE Marketing”, Zhang, and Huo teach:
The method of claim 1, wherein determining the coordinates of the corners of the planar surface comprises:
determining a filtered set of straight lines from a plurality of edges in a contour image extracted from the segmentation mask (Huo [Pg 8 Par 3] then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form, that is using distance p and angle theta to represent the straight line, a plurality of said parameters correspond to a plurality of straight lines passing through the edge point of said lane solid line; then, the intersection analysis according to the first straight line parameter and the second straight line parameter is to convert a plurality of straight line families passing through a plurality of lane solid line edge points into a plurality of lines in a parameter space according to a plurality of the parameters, wherein the straight line family corresponds to the line in the parameter space one by one;);
determining intersections of the filtered set of straight lines as candidate corner points (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space.); and
verifying that a shape formed by the candidate corner points form a valid polygon (Huo [Pg 8 Par 7] Through edge detection, feature analysis, shape processing and smooth fitting, the invention achieves the technical effect of providing accurate tyre connecting line for subsequent line pressing detection.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Regarding claim 18.
Long and “LYFE Marketing” teach:
The non-transitory computer readable medium of claim 17, wherein determining the coordinates of the corners of the planar surface comprises:
Long and “LYFE Marketing” fail to teach:
detecting a plurality of edges of the outer contour utilizing an edge detection algorithm (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
determining sets of one or more straight lines along the outer contour from the plurality of edges utilizing a Hough transform with a plurality of threshold intersection values (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.);
determining a filtered set of straight lines from the sets of one or more straight lines according to a distance threshold and an angle threshold (Huo [Pg 8 Par 3] then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form, that is using distance p and angle theta to represent the straight line, a plurality of said parameters correspond to a plurality of straight lines passing through the edge point of said lane solid line; then, the intersection analysis according to the first straight line parameter and the second straight line parameter is to convert a plurality of straight line families passing through a plurality of lane solid line edge points into a plurality of lines in a parameter space according to a plurality of the parameters, wherein the straight line family corresponds to the line in the parameter space one by one;); and
determining the coordinates of the corners from intersections of the filtered set of straight lines (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space.).
Zhang teaches:
detecting a plurality of edges of the outer contour utilizing an edge detection algorithm (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
Huo teaches:
determining sets of one or more straight lines along the outer contour from the plurality of edges utilizing a Hough transform with a plurality of threshold intersection values (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.);
determining a filtered set of straight lines from the sets of one or more straight lines according to a distance threshold and an angle threshold (Huo [Pg 8 Par 3] then, performing the linear equation conversion, obtaining a plurality of the first linear parameter and the second linear parameter, wherein the first linear parameter and the second linear parameter are represented as polar coordinate form, that is using distance p and angle theta to represent the straight line, a plurality of said parameters correspond to a plurality of straight lines passing through the edge point of said lane solid line; then, the intersection analysis according to the first straight line parameter and the second straight line parameter is to convert a plurality of straight line families passing through a plurality of lane solid line edge points into a plurality of lines in a parameter space according to a plurality of the parameters, wherein the straight line family corresponds to the line in the parameter space one by one;); and
determining the coordinates of the corners from intersections of the filtered set of straight lines (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU) and Yang et al. (US 20220084307).
Regarding claim 6.
Long and “LYFE Marketing” teach:
The method of claim 1, wherein determining the coordinates of the corners of the planar surface comprises:
determining the coordinates of the corners of the planar surface from the candidate corner points in response to determining that the intersection-over-union metric is at least a threshold intersection-over-union value (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
Long and “LYFE Marketing” fail to teach:
generating an intersection-over-union metric between the segmentation mask and a shape formed by candidate corner points of the planar surface (Yang [0050] For example, the matching algorithm mainly adopts methods such as the contour mask shape matching based on key features and segmentation mask, the intersection over union (IOU) matching, the geometric feature (such as length-width ration of a bounding rectangle) matching and the histogram matching. In addition, the measurement of the similarity in the present disclosure may be achieved through a cosine distance.); and
Yang teaches:
generating an intersection-over-union metric between the segmentation mask and a shape formed by candidate corner points of the planar surface (Yang [0050] For example, the matching algorithm mainly adopts methods such as the contour mask shape matching based on key features and segmentation mask, the intersection over union (IOU) matching, the geometric feature (such as length-width ration of a bounding rectangle) matching and the histogram matching. In addition, the measurement of the similarity in the present disclosure may be achieved through a cosine distance.); and
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and “LYFE Marketing” with Yang. Finding intersections and using segmentation, as in Yang, would benefit the Long and “LYFE Marketing” teachings by being able to use different methods to find certain points such as corners. Additionally, this is the application of a known technique, finding intersections and using segmentation, to yield predictable results.
Claim(s) 8 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU) and Hu et al. (US 20210350187).
Regarding claim 8.
Long and “LYFE Marketing” teach:
The method of claim 7, further comprising:
Long and “LYFE Marketing” alone fail to teach:
determining a subset of filtered digital images of the training dataset comprising a first set of digital images with planar surfaces and a second set of digital images without planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
training the classifier model utilizing the subset of filtered digital images of the training dataset to classify digital images including planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).
Long, “LYFE Marketing”, and Hu teach:
determining a subset of filtered digital images of the training dataset comprising a first set of digital images with planar surfaces and a second set of digital images without planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
training the classifier model utilizing the subset of filtered digital images of the training dataset to classify digital images including planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and “LYFE Marketing” with Hu. Training a neural network, as in Hu, would benefit the Long and “LYFE Marketing” teachings training a neural network to be able to be more accurate. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Regarding claim 9.
Long, “LYFE Marketing”, and Hu teach:
The method of claim 8, wherein generating the modified training dataset comprises:
generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset indicating whether the plurality of filtered digital images include planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and “LYFE Marketing” with Hu. Training a neural network, as in Hu, would benefit the Long and “LYFE Marketing” teachings training a neural network to be able to be more accurate. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Claim(s) 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of Hu et al.( US 20210350187).
Regarding claim 10.
Long teaches:
A method comprising:
receiving a training dataset comprising a plurality of filtered digital images with a specified set of visual attributes (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.).
training a segmentation neural network using the modified training dataset to generate a trained segmentation neural network that generates segmentation masks for planar surfaces in digital images (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.).
Long alone fails to teach:
classifying, utilizing a classifier model, the plurality of filtered digital images to indicate whether the plurality of filtered digital images comprise planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.)
generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset indicating whether the plurality of filtered digital images include planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).);
generating a modified training dataset by filtering the training dataset to a subset of digital images of the plurality of filtered digital images classified as comprising planar surfaces (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset indicating whether the plurality of filtered digital images include planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).); and
Long and Hu teaches:
classifying, utilizing a classifier model, the plurality of filtered digital images to indicate whether the plurality of filtered digital images comprise planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset indicating whether the plurality of filtered digital images include planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).);
generating a modified training dataset by filtering the training dataset to a subset of digital images of the plurality of filtered digital images classified as comprising planar surfaces (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset indicating whether the plurality of filtered digital images include planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).); and
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with Hu. Training a neural network, as in Hu, would benefit the Long teachings by training a neural network to be able to be more accurate. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Regarding claim 11.
Long and Hu teach:
The method of claim 10, further comprising:
determining a subset of digital images of the plurality of filtered digital images comprising a first set of digital images with planar surfaces and a second set of digital images without planar surfaces(Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
training the classifier model to classify the first set of digital images as planar surface images and the second set of digital images as non-planar surface images (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with Hu. Training a neural network, as in Hu, would benefit the Long teachings by training a neural network to be able to be more accurate. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Regarding claim 12.
Long and Hu teach:
The method of claim 10, wherein classifying the plurality of filtered digital images comprises:
generating, utilizing the classifier model, a plurality of predicted classifications for the plurality of filtered digital images in the training dataset as planar surface images or non-planar surfaces (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
filtering the training dataset to the subset of digital images comprising planar surface images in response to determining that classification confidence scores of predicted classifications for the subset of digital images of the plurality of filtered digital images meets a confidence threshold (Long [0075] The computer-generated training datasets may comprise photo-realistic renderings which provide high-quality depth information, thus supporting the network to make geometrically consistent predictions.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long with Hu. Training a neural network, as in Hu, would benefit the Long teachings by training a neural network to be able to be more accurate. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of Hu et al.( US 20210350187), Tan et al. (US 20220101048), and “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU).
Regarding claim 13.
Long and Hu teach:
The method of claim 10, wherein generating the modified training dataset comprises:
Long and Hu fail to teach:
annotating the subset of digital images to mark the planar surfaces (Tan [0013] In accordance with these implementations, based on selection of the synthetic 2D image, the annotation transfer component can further transfer the ground truth data to the native 2D image using a subset of the different projection parameters used to generate the synthetic 2D image, resulting in generation of an annotated native 2D image. The training module can additionally or alternatively employ the native 2D image and the annotate); and
generating, for the modified training dataset, modified digital images by augmenting the subset of digital images with a multiply blending filter or perspective transformation modifications to modify the planar surfaces in the subset of digital images (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.).
Tan teaches:
annotating the subset of digital images to mark the planar surfaces (Tan [0013] In accordance with these implementations, based on selection of the synthetic 2D image, the annotation transfer component can further transfer the ground truth data to the native 2D image using a subset of the different projection parameters used to generate the synthetic 2D image, resulting in generation of an annotated native 2D image. The training module can additionally or alternatively employ the native 2D image and the annotate); and
“LYFE Marketing” teaches:
generating, for the modified training dataset, modified digital images by augmenting the subset of digital images with a multiply blending filter or perspective transformation modifications to modify the planar surfaces in the subset of digital images (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and Hu with Tan and “LYFE Marketing”. Annotating subsets of images and having specific images that align with what the neural network should be doing, as in Tan and “LYFE Marketing”, would benefit the Long and Hu teachings by training a neural network to be able to be more accurate based on data sets that you control. Additionally, this is the application of a known technique, training a neural network with specific data, to yield predictable results.
Claim(s) 14 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of Hu et al.( US 20210350187), and “LYFE Marketing” et al.( https://www.youtube.com/watch?v=DOVCQdSxLOU).
Regarding claim 14.
Long and Hu teach:
The method of claim 10, further comprising:
generating, utilizing the trained segmentation neural network, a segmentation mask of a planar surface in a digital image (Long [0030] Image segmentation component 130 may identify image segmentation information for the 2D image. Image segmentation component 130 may also extract object information using a mask regional convolutional neural network (Mask R-CNN). The image segmentation information is based on the object information.);
Long and Hu fail to teach:
determining coordinates of corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.); and
generating a modified digital image comprising a digital asset inserted onto the planar surface by utilizing a transformation matrix to perform a perspective transformation on the digital asset according to the coordinates of the corners (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
“LYFE Marketing” teaches:
determining coordinates of corners of the planar surface (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.); and
generating a modified digital image comprising a digital asset inserted onto the planar surface by utilizing a transformation matrix to perform a perspective transformation on the digital asset according to the coordinates of the corners (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and Hu with “LYFE Marketing”. Being able to change different places in an image with a different image using ai, as in “LYFE Marketing”, would benefit the Long and Hu teachings by using a neural network to be able to change aspects of an image. Additionally, this is the application of a known technique, being able to change different places in an image with a different image using ai, to yield predictable results.
Regarding claim 20.
Long with Hu teach:
The non-transitory computer readable medium of claim 17, wherein the operations further comprise:
generating a training dataset comprising a plurality of modified digital images by: classifying, utilizing a classifier model, a plurality of digital images as planar surface images or non-planar surface images (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.) (Hu [0057] The planar image information includes first-dimensional image information and second-dimensional image information. In an example embodiment of the disclosure, the image classification model extracts the first image feature from the 3D image. The image classification model is obtained by training a neural network model by using a 3D sample image annotated with an expected classification result. A neural network may be a convolutional neural network (CNN), an artificial neural network (ANN), a deep neural network (DNN), or the like. This is not limited in an example embodiment of the disclosure. A machine learning algorithm used during training of a machine learning model may be a back-propagation (BP) algorithm, a faster regions with convolutional neural network (faster RCNN) algorithm, or the like. This is not limited in an example embodiment of the disclosure. For example structures of the image classification model and functions implemented by the structures, reference may be made to FIG. 1 to FIG. 6 and related descriptions. Details are not described herein again.); and
training, utilizing the training dataset, the segmentation neural network to detect planar surfaces in digital images (Long [0013] Therein, a deep neural network for depth prediction and train the network in a geometry-aware manner is provided. The depth prediction is further coupled with a segmentation-based depth adjustment process to enable effective depth prediction for view synthesis. A depth-based synthesis model is further created, that allows the generation of novel views along the desired camera path.).
Long and Hu fails to teach:
generating the plurality of modified digital images by augmenting the planar surface images utilizing a multiply blending filter or a perspective transformation (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
“LYFE Marketing” teaches:
generating the plurality of modified digital images by augmenting the planar surface images utilizing a multiply blending filter or a perspective transformation (“LYFE Marketing” [1:20-2:00] The video shows that the corners are detected and moved onto the surface. This way it allows for any image to be moved onto another surface such as a computer screen. Canva also has other ones you can use.); and
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long and Hu with “LYFE Marketing”. Being able to change different places in an image with a different image using ai, as in “LYFE Marketing”, would benefit the Long and Hu teachings by using a neural network to be able to change aspects of an image. Additionally, this is the application of a known technique, being able to change different places in an image with a different image using ai, to yield predictable results.
Claim(s) 15 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (US 11017586) in view of Hu et al. (US 20210350187), “LYFE Marketing” et al. (https://www.youtube.com/watch?v=DOVCQdSxLOU), Zhang et al. (CN 110717852), and Huo et al. (CN 117037083).
Regarding claim 15.
Long, Hu, and “LYFE Marketing” teach:
The method of claim 14, wherein determining the coordinates of the corners of the planar surface comprises:
determining the coordinates of the corners from intersections of the plurality of straight lines (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to. Certain surface can have straight lines).
Long, Hu, and “LYFE Marketing” fail to teach:
generating a contour image comprising an outer contour from the segmentation mask with a solid fill color inside the outer contour (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
generating, utilizing an edge detection algorithm, an edge map by detecting a plurality of edges in the contour image (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
determining a plurality of straight lines along the outer contour from the plurality of edges in the edge map (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.); and
Zhang teaches:
generating a contour image comprising an outer contour from the segmentation mask with a solid fill color inside the outer contour (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
generating, utilizing an edge detection algorithm, an edge map by detecting a plurality of edges in the contour image (Zhang [Pg 14 Par 5] firstly loading the RGB colour image, then carrying out gradation for the image, then performing Gauss filtering to remove noise on the image, then using the Canny operator to do the edge detection to obtain the contour of the image, and the outline of the different areas is numbered. Finally, each region of the image segmentation and image of colour filling, and filling fused with the original image to obtain the final segmented image,);
Huo teaches:
determining a plurality of straight lines along the outer contour from the plurality of edges in the edge map (Huo [Pg 8 Par 3] he straight line with any angle and length in the image can be detected by using the Hough transformation to detect the straight line so as to reach the technical effect of good robustness for the noise and the incomplete lane solid line.); and
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, Hu, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, Hu, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Regarding claim 16.
Long, Hu, “LYFE Marketing”, Zhang, and Huo teach:
The method of claim 15, wherein determining the coordinates of the corners of the planar surface further comprises:
determining candidate corner points from the intersections of the plurality of straight lines (Huo [Pg 8 Par 3] Then, an accumulator is created to count the intersection of the lines in the parameter space.);
determining a shape formed by the candidate corner points (Huo [Pg 8 Par 7] Through edge detection, feature analysis, shape processing and smooth fitting, the invention achieves the technical effect of providing accurate tyre connecting line for subsequent line pressing detection.);
generating an intersection-over-union metric between the segmentation mask and the shape formed by the candidate corner points (Huo [Pg 8 Par 7] Through edge detection, feature analysis, shape processing and smooth fitting, the invention achieves the technical effect of providing accurate tyre connecting line for subsequent line pressing detection.); and
determining the coordinates of the corners in response to determining that the intersection-over-union metric is at least a threshold intersection-over-union value (“LYFE Marketing” [1:20-2:00] The video shows the transformation once it is done and creates a modified image in which the corners are conformed to. Certain surface can have straight lines).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Long, Hu, “LYFE Marketing”, and Zhang with Huo. Detecting certain edges and being able to create or detect straight lines, as in Huo, would benefit the Long, Hu, “LYFE Marketing”, and Zhang teachings by using edges to figure out straight lines. Additionally, this is the application of a known technique, detecting certain edges and being able to create or detect straight lines, to yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENIS VASILIY MINKO whose telephone number is (571)270-5226. The examiner can normally be reached Monday-Thursday 8:30-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENIS VASILIY MINKO/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights