DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submissions, filed on June 03rd, 2026, have been entered after the RCE filed on July 07th, 2026, the amendment of the submissions have been considered and entered into.
Status of Claims
Claims 1-3, 5-11, 13-14 and 17-24 are pending, claims 1-2, 7-11 and 17-20 have been amended, claims 21-24 have been added. Claims 1-3, 5-11, 13-14 and 17-24 remains rejected.
Response to Argument(s)
In view of the Amendments to independent claims 1, 7 and 20, the previously applied prior art rejections are withdrawn. Applicants’ arguments are rendered moot in view of the new grounds of rejection set forth below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 7-8, 20-21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Suryanarayana M. Muddala et. al. (“Spatio-Temporal Consistent Depth-Image-Based Rendering using Layered Depth Image and Inpainting, Feb. 2016, EURASIP Journal on Image and Video Processing, Vol. 2016, Article Number 9” hereinafter as “Muddala”) and Xiaoyan Sun et. al. (“US 2008/0238942 A1” hereinafter as “Sun”).
Regarding claim 1, Muddala teaches a system comprising: a computing device, comprising: one or more processors; a memory; and a non-transitory computer readable medium having instructions stored thereon that when executed by a processor cause performance of a set of functions, wherein the set of functions comprises (Page 3 of Section 1.1, last par., discloses the area of the invention is in computer vision which indicates the use of a computer, which includes a processor to execute instructions [functions] stored in a memory and a RAM/ROM [non-transitory computer readable medium] to execute out these functions of the invention stored in a program): receiving an image from an image capture device (Fig. 3 illustrates the input to the processing is an image which is received from an camera [image capture device] according to Fig. 1), wherein the image represents a scene (Section 2, 1st Par., discloses “construct a statis scene sprite over time and fill the holes by using a depth-based inpainting method”) that includes a subject and a distractor that is visible in the image (Section 2.4.3 discloses “the threshold for the classification is selected from the previously generated threshold image…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects” wherein, the background object is analogous to the recited subject being a target object, and the artifact is analogous to the recited distractor); identifying the distractor for removal from the image by inpainting (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”); determining a mask for the image (FIG. 3 illustrated a mask generation for the input image), wherein the mask comprises (i) an inpainting region (Moreover, FIG. 3 shows that the mask includes both regions of the occlusion [shaded white areas] and regions not to be inpainted [other than the shaded white areas, which is the black area] including occlusion areas and disocclusion areas such as shown in FIG. 4, wherein the occlusion areas being inpainting region, according to Section 2.7), and (ii) a non-inpainting region that is not to be inpainted (Moreover, FIG. 3 shows that the mask includes both regions of the occlusion [shaded white areas] and regions not to be inpainted [other than the shaded white areas, which is the black area] including occlusion areas and disocclusion areas such as shown in FIG. 4), wherein the non-inpainting region includes the subject (The recited “non-inpainting region” shares the same scope with Muddala’s disocclusion areas, since Muddala’s disocclusion areas as disclosed in Muddala’s abstract being used a method that would result in no disocclusions to be appearing in the rendered data; moreover, Section 2.1.1, 1st Par., discloses “a disocclusion appears to the right of the foreground object”; Section 1, 4th Par., discloses “disocclusions occur at the borders of the object” indicating the disocclusion areas include the subject part); determining a depth representation comprising a pixelwise depth estimation for the scene represented by the image (Fig.3 illustrates a depth representation “Depth(1…N_t)” which represents the depth estimate for each pixel [pixelwise depth estimation]) wherein the subject has a first depth estimation and the distractor has a second depth estimation that is different than the first depth estimation (Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates), wherein the inpainting region in the mask comprises pixels (Page 7 of section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]) that correspond to the first depth estimation of the subject (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the background object is analogous to the recited first depth estimation) and pixels that correspond to the second depth estimation of the distractor (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the foreground object is analogous to the recited second depth estimation); refining the inpainting region (Section 2.1.3 discloses the occlusion identification to correctly identify background and foreground and inpaint the occluded areas [refining as claimed] including the mislabeled background and foreground points with overlapped labelled points in overlapped DDP as previously discussed in Page 7 of section 2.2); and inpainting the image in accordance with the refined inpainting region (The inpainting [hole-filling] such as illustrated in FIG. 3 to fille in the occluded region, example of the occluded image is shown in FIG. 5.d; the process of the filling is further disclosed in 2.4.3 wherein the filling happens so that only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects; therefore, the inpainting process here is to make sure that the foreground region is excluded in the process so that the foreground region is not to be having artifacts [not needed for filling of to be inpainted]; in another sense, the overlapped DDP is now corrected labeled and not needed to be inpainted further in the corrected patch as disclosed in 2.4.3) such that the pixels that correspond to the first depth estimation of the subject are not inpainted (Section 2.4.3 discloses “to avoid the patch selection from the foreground during the inpainting process, the source region is classified into the foreground-background locally. The threshold for the classification is selected from the previously generated threshold image ZAS…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects”).
PNG
media_image1.png
128
218
media_image1.png
Greyscale
Muddala’s Fig. 3 occlusion mask includes regions to be inpainted.
PNG
media_image2.png
424
1134
media_image2.png
Greyscale
Muddala’s Figure 4 of the overlapped DDP wherein the pixel-labeled background is labeled as foreground in the disocclusion area [non-inpainting region]
However, Muddala does not explicitly teach an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Sun teaches an inpainting region that outlines the distractor (Par. [0017] discloses “the missing region contains part of a lead image…can abstract structure information (e.g., the object’s border or outline information…that will be inpainted in the missing region)”; Par. [0052] discloses “segmentation to differentiate foreground-the target object-and background (the pre-existing image)…so that the target object borders and the image contents they contain take precedence over portions of overlapping background” wherein, the region to be inpainted include outlines of the region between background and foreground); wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject (Par. [0044] discloses “known pixels belonging to the border (i.e., the structure) are not counted in the confidence computation” wherein, Par. [0005] discloses “improve inpainting capability by indicating the source region, depth information, and important salient structure” wherein, the confidence computation is part of the inpainting process, wherein to omit/not counting the pixels of the subject that are known [analogous to the first depth estimation part]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Sun of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Wherein having Muddala’s system of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed. Since both Muddala and Sun perform image inpainting using background-foreground regions. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Sun’s system improves inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed (see Sun’s Par. [0005]).
Regarding claim 7, Muddala teaches a method comprising: receiving an image from an image capture device (Fig. 3 illustrated the input to the processing is an image which is received from an camera [image capture device] according to Fig. 1), wherein the image represents a scene (Section 2, 1st Par., discloses “construct a statis scene sprite over time and fill the holes by using a depth-based inpainting method”) that includes a subject and a distractor that is visible in the image (Section 2.4.3 discloses “the threshold for the classification is selected from the previously generated threshold image…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects” wherein, the background object is analogous to the recited subject being a target object, and the artifact is analogous to the recited distractor); determining a mask for the image (FIG. 3 illustrated a mask generation for the input image), wherein the mask comprises (i) an inpainting region (The mask generated in FIG. 3 includes occlusion region [region to be inpainted]), and (ii) a non-inpainting region that is not to be inpainted (Moreover, FIG. 3 shows that the mask includes both regions of the occlusion [shaded white areas] and regions not to be inpainted [other than the shaded white areas, which is the black area] including occlusion areas and disocclusion areas such as shown in FIG. 4), wherein the non-inpainting region includes the subject (The recited “non-inpainting region” shares the same scope with Muddala’s disocclusion areas, since Muddala’s disocclusion areas as disclosed in Muddala’s abstract being used a method that would result in no disocclusions to be appearing in the rendered data; moreover, Section 2.1.1, 1st Par., discloses “a disocclusion appears to the right of the foreground object”; Section 1, 4th Par., discloses “disocclusions occur at the borders of the object” indicating the disocclusion areas include the subject part); determining a depth representation comprising a pixelwise depth estimation for a scene represented by the image (Fig.3 illustrates a depth representation “Depth(1…N_t)” which represents the depth estimate for each pixel [pixelwise depth estimation]) wherein the subject has a first depth estimation and the distractor has a second depth estimation that is different than the first depth estimation (Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates), wherein the inpainting region in the mask comprises pixels (Page 7 of section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]) that correspond to the first depth estimation of the subject (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the background object is analogous to the recited first depth estimation) and pixels that correspond to the second depth estimation of the distractor (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the foreground object is analogous to the recited second depth estimation); refining the inpainting region (Section 2.1.3 discloses the occlusion identification to correctly identify background and foreground and perform inpainting on the occluded areas [refining as claimed] including the mislabeled background and foreground points with overlapped labelled points in overlapped DDP as previously discussed in Page 7 of section 2.2; As shown in FIG. 4 and disclosed in Page 7 of section 2.2, the overlapped DDP include the mislabeled of the background point and the foreground point into the disocclusion area [non-inpainting region]); and inpainting the image in accordance with the refined inpainting region (The inpainting [hole-filling] such as illustrated in FIG. 3 to fille in the occluded region, example of the occluded image is shown in FIG. 5.d; the process of the filling is further disclosed in 2.4.3 wherein the filling happens so that only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects; therefore, the inpainting process here is to make sure that the foreground region is excluded in the process so that the foreground region is not to be having artifacts [not needed for filling of to be inpainted]; in another sense, the overlapped DDP is now corrected labeled and not needed to be inpainted further in the corrected patch as disclosed in 2.4.3) such the pixels that correspond to the first depth estimation of the subject are not inpainted (Section 2.4.3 discloses “to avoid the patch selection from the foreground during the inpainting process, the source region is classified into the foreground-background locally. The threshold for the classification is selected from the previously generated threshold image ZAS…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects”).
However, Muddala does not explicitly teach an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Sun teaches an inpainting region that outlines the distractor (Par. [0017] discloses “the missing region contains part of a lead image…can abstract structure information (e.g., the object’s border or outline information…that will be inpainted in the missing region)”; Par. [0052] discloses “segmentation to differentiate foreground-the target object-and background (the pre-existing image)…so that the target object borders and the image contents they contain take precedence over portions of overlapping background” wherein, the region to be inpainted include outlines of the region between background and foreground); wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject (Par. [0044] discloses “known pixels belonging to the border (i.e., the structure) are not counted in the confidence computation” wherein, Par. [0005] discloses “improve inpainting capability by indicating the source region, depth information, and important salient structure” wherein, the confidence computation is part of the inpainting process, wherein to omit/not counting the pixels of the subject that are known [analogous to the first depth estimation part]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Sun of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Wherein having Muddala’s method of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed. Since both Muddala and Sun perform image inpainting using background-foreground regions. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Sun’s system improves inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed (see Sun’s Par. [0005]).
Regarding claim 8, Muddala in view of Sun teaches the method of claim 7, Muddala teaches wherein refining the inpainting region is performed concurrently with determining the depth representation for the image (As shown in Muddala’s FIG. 3, the Fig. 3’s caption further discloses the texture map processing and the depth map processing is happened parallel, or concurrently).
Regarding claim 20, Muddala teaches a non-transitory computer readable medium having instructions stored thereon that when executed by a processor cause performance of a set of functions, wherein the set of functions comprises (Page 3, last par., discloses the area of the invention is in computer vision which indicates the use of a computer, which includes a processor to execute instructions [functions] stored in a memory and a RAM/ROM [non-transitory computer readable medium] to execute out these functions of the invention stored in a program): receiving an image from an image capture device (Fig. 3 illustrates the input to the processing is an image which is received from an camera [image capture device] according to Fig. 1), wherein the image represents a scene (Section 2, 1st Par., discloses “construct a statis scene sprite over time and fill the holes by using a depth-based inpainting method”) that includes a subject and a distractor that is visible in the image (Section 2.4.3 discloses “the threshold for the classification is selected from the previously generated threshold image…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects” wherein, the background object is analogous to the recited subject being a target object, and the artifact is analogous to the recited distractor); identifying the distractor for removal from the image by inpainting (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”); determining a mask for the image (FIG. 3 illustrated a mask generation for the input image), wherein the mask comprises (i) an inpainting region (The mask generated in FIG. 3 includes occlusion region [region to be inpainted]), and (ii) a non-inpainting region that is not to be inpainted (Moreover, FIG. 3 shows that the mask includes both regions of the occlusion [shaded white areas] and regions not to be inpainted [other than the shaded white areas, which is the black area] including occlusion areas and disocclusion areas such as shown in FIG. 4) , wherein the non-inpainting region includes the subject (The recited “non-inpainting region” shares the same scope with Muddala’s disocclusion areas, since Muddala’s disocclusion areas as disclosed in Muddala’s abstract being used a method that would result in no disocclusions to be appearing in the rendered data; moreover, Section 2.1.1, 1st Par., discloses “a disocclusion appears to the right of the foreground object”; Section 1, 4th Par., discloses “disocclusions occur at the borders of the object” indicating the disocclusion areas include the subject part); determining a depth representation comprising a pixelwise depth estimation for a scene represented by the image (Fig.3 illustrates a depth representation “Depth(1…N_t)” which represents the depth estimate for each pixel [pixelwise depth estimation]) wherein the subject has a first depth estimation and the distractor has a second depth estimation that is different than the first depth estimation (Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates), wherein the inpainting region in the mask comprises pixels (Page 7 of section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]) that correspond to the first depth estimation of the subject (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the background object is analogous to the recited first depth estimation) and pixels that correspond to the second depth estimation of the distractor (Page 7 of Section 2.2 discloses pixels in the DDP [constituent of the mask] includes background and foreground labeling [two features]; Page 7 of Section 2.2, discloses the background and the foreground regions have different depth estimates; wherein, the pixels in DDP of the foreground object is analogous to the recited second depth estimation) ; refining the inpainting region (Section 2.1.3 discloses the occlusion identification to correctly identify background and foreground and inpaint the occluded areas [refining as claimed] including the mislabeled background and foreground points with overlapped labelled points in overlapped DDP as previously discussed in Page 7 of section 2.2); and inpainting the image in accordance with the refined inpainting region (The inpainting [hole-filling] such as illustrated in FIG. 3 to fille in the occluded region, example of the occluded image is shown in FIG. 5.d; the process of the filling is further disclosed in 2.4.3 wherein the filling happens so that only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects; therefore, the inpainting process here is to make sure that the foreground region is excluded in the process so that the foreground region is not to be having artifacts [not needed for filling of to be inpainted]; in another sense, the overlapped DDP is now corrected labeled and not needed to be inpainted further in the corrected patch as disclosed in 2.4.3) such that the pixels that correspond to the first depth estimation of the subject are not inpainted (Section 2.4.3 discloses “to avoid the patch selection from the foreground during the inpainting process, the source region is classified into the foreground-background locally. The threshold for the classification is selected from the previously generated threshold image ZAS…the block matching is performed with only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects”).
PNG
media_image1.png
128
218
media_image1.png
Greyscale
Muddala’s Fig. 3 occlusion mask includes regions to be inpainted.
PNG
media_image2.png
424
1134
media_image2.png
Greyscale
Muddala’s Figure 4 of the overlapped DDP wherein the pixel-labeled background is labeled as foreground in the disocclusion area [non-inpainting region]
However, Muddala does not explicitly teach an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Sun teaches an inpainting region that outlines the distractor (Par. [0017] discloses “the missing region contains part of a lead image…can abstract structure information (e.g., the object’s border or outline information…that will be inpainted in the missing region)”; Par. [0052] discloses “segmentation to differentiate foreground-the target object-and background (the pre-existing image)…so that the target object borders and the image contents they contain take precedence over portions of overlapping background” wherein, the region to be inpainted include outlines of the region between background and foreground); wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject (Par. [0044] discloses “known pixels belonging to the border (i.e., the structure) are not counted in the confidence computation” wherein, Par. [0005] discloses “improve inpainting capability by indicating the source region, depth information, and important salient structure” wherein, the confidence computation is part of the inpainting process, wherein to omit/not counting the pixels of the subject that are known [analogous to the first depth estimation part]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Sun of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
Wherein having Muddala’s system of having an inpainting region that outlines the distractor; wherein refining the inpainting region comprises omitting the pixels that correspond to the first depth estimation of the subject.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed. Since both Muddala and Sun perform image inpainting using background-foreground regions. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Sun’s system improves inpainting capability by indicating the source region, depth information and important salient structure with high-level knowledge employed, (see Sun’s Par. [0005]).
Regarding claim 21, Muddala in view of Sun teaches the system of claim 1, Muddala teaches wherein identifying the distractor for removal from the image by inpainting (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”) is based on a user selection of a region of the image (Section 1, 1st Par., discloses “the FTV application offers users the ability to look around the scene by generating new views” wherein, the region of the image is viewed by user’s selection for such processing to happen accordingly).
Regarding claim 23, Muddala in view of Sun teaches the system of claim 7, Muddala teaches wherein identifying the distractor for removal from the image by inpainting (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”) is based on a user selection of a region of the image (Section 1, 1st Par., discloses “the FTV application offers users the ability to look around the scene by generating new views” wherein, the region of the image is viewed by user’s selection for such processing to happen accordingly).
Claims 2-3 and 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Suryanarayana M. Muddala et. al. (“Spatio-Temporal Consistent Depth-Image-Based Rendering using Layered Depth Image and Inpainting, Feb. 2016, EURASIP Journal on Image and Video Processing, Vol. 2016, Article Number 9” hereinafter as “Muddala”) and Xiaoyan Sun et. al. (“US 2008/0238942 A1” hereinafter as “Sun”) in view of Peter Amon et. al. (“US 2022/0076117 A1” hereinafter as “Amon”).
Regarding claim 2, Muddala in view of Sun teaches the system of claim 1, wherein refining the inpainting region comprises applying the image and the mask to output a refined mask comprising the refined inpainting region (As discussed above in claim 1, by using the input and the mask of FIG. 3 and Fig. 4 a refined mask is output including refined respective inpainting region [according to section 2.4.3 wherein the refined region including the generated threshold image]).
However, Muddala in view of Sun does not explicitly teach applying a machine learning model to the image and the mask.
Amon teaches applying a machine learning model to the image and the mask (paragraphs [0010]-[0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image, including the image and the mask of the objects according to par. [0011]; therefore, the process of Muddala as discussed is to output a result include processing of the detection of the object and classify the object regions as foreground or background which is now taught by Amon to include using machine learning model).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where applying a machine learning model to the image and the mask.
Wherein having Muddala’s system having where applying a machine learning model to the image and the mask.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
Regarding claim 3, Muddala in view of Sun and Amon teaches the system of claim 2, Muddala teaches wherein the computing device and the machine learning model are part of a server system (Muddala, as discussed above, discloses the processing as part of a computer vision area, hence, includes using of a computer [server system] such as disclosed in page 3, last par.).
Regarding claim 9, Muddala in view of Sun teaches the method of claim 8, Muddala teaches wherein refining the inpainting region concurrently with determining the depth representation for the image comprises applying a model to the image and to the mask, wherein the model outputs a refined mask comprising the refined inpainting region (By using the input and the mask of FIG. 3 and Fig. 4 a refined mask is output including refined respective inpainting region [according to section 2.4.3 wherein the refined region including the generated threshold image]).
However, Muddala in view of Sun does not explicitly teach applying a machine learning model to the image and the mask, and the model being a machine learning model.
Amon teaches applying a machine learning model to the image and the mask (paragraphs [0010]-[0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image, including the image and the mask of the objects according to par. [0011]; the whole process of Muddala as discussed above in claims 7 and 8 to output a result include processing of the detection of the object and classify the object regions as foreground or background which is now taught by Amon to include using machine learning model) , and the model being a machine learning model (paragraphs [0010]-[0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where applying a machine learning model to the image and the mask, and the model being a machine learning model.
Wherein having Muddala’s method of having where applying a machine learning model to the image and the mask, and the model being a machine learning model.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
Regarding claim 10, Muddala in view of Sun and Amon teaches the method of claim 9.
However, Muddala in view of Sun does not explicitly teach wherein the machine learning model determines both the depth representation and the refined inpainting region.
Amon discloses wherein the machine learning model determines both the depth representation and the refined inpainting region (Amon teaches using machine learning to determine depth representation and inpainting according to Amon’s par. [0047]; moreover, since the objects are determined by using a machine learning as discussed in Amon’s par. [0016] therefore, the objects identified in Amon using the machine learning model is used for both the depth representation of the object and the refined threshold image of Muddala such as shown in Muddala’s Fig. 3 and Fig. 4).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where wherein the machine learning model determines both the depth representation and the refined inpainting region.
Wherein having Muddala’s method having wherein the machine learning model determines both the depth representation and the refined inpainting region.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
Regarding claim 11, Muddala in view of Sun teaches the method of claim 7, Muddala teaches wherein refining the inpainting region comprises applying the image and the mask to output a refined mask comprising the refined inpainting region (By using the input and the mask of FIG. 3 and Fig. 4 a refined mask is output including refined respective inpainting region [according to section 2.4.3 wherein the refined region including the generated threshold image]).
However, Muddala in view of Sun does not explicitly teach applying a machine learning model to the image and the mask.
Amon teaches applying a machine learning model to the image and the mask. (paragraphs [0010]-[0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image, including the image and the mask of the objects according to par. [0011]; therefore, the whole process of Muddala as discussed above in claim 7 to output a result include processing of the detection of the object and classify the object regions as foreground or background which is now taught by Amon to include using machine learning model).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where applying a machine learning model to the image and the mask.
Wherein having Muddala’s method of having where applying a machine learning model to the image and the mask.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
Claims 5-6 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Suryanarayana M. Muddala et. al. (“Spatio-Temporal Consistent Depth-Image-Based Rendering using Layered Depth Image and Inpainting, Feb. 2016, EURASIP Journal on Image and Video Processing, Vol. 2016, Article Number 9” hereinafter as “Muddala”) in view of Xiaoyan Sun et. al. (“US 2008/0238942 A1” hereinafter as “Sun”) further in view of Peter Amon et. al. (“US 2022/0076117 A1” hereinafter as “Amon”) and Nikita Dvornik et. al. (“On the Importance of Visual Context for Data Augmentation in Scene Understanding, June 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 61” hereinafter as “Dvornik”).
Regarding claim 5, Muddala in view of Sun and Amon teaches the system of claim 2.
However, Muddala in view of Sun does not explicitly teach the set of functions further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
Amon teaches the set of functions further comprising: obtaining a plurality of training images (Amon, par. [0021] discloses the training include obtaining a number of training images being fed to the feature extractor); adding an image feature to each of the training images (Amon, par. [0021] discloses the feature extractor to extract features from the training images; furthermore, par. [0043] discloses the training images may include images that have same scene is combined or augmented with objects or objects of predetermined types, in other word, the object being extracted the features can be added into the training images to create augmented images to be part of the training images); creating a plurality of training masks corresponding to the plurality of training images (Amon, par. [0043] discloses generation of training images include augmenting the starting image with another instance of an object of a predetermined type, and for a plurality of images would include the use of a plurality of objects of predetermined types; moreover, these instances are represented by pixel map according to par. [0047] which is analogous to “mask”), wherein each training mask comprises an image feature region comprising an initial outline of the added feature (as disclosed in Amon’s par. [0043], the portion of the another instance of object can be understood as the image feature including an initial of the added feature as claimed; moreover, these objects are represented by object outline according to [0132]); augmenting each feature region by adjusting the initial outline of the added feature (Amon, par. [0043-0045] discloses the training images are generated synthetically by augmenting the instances of objects into the starting image, the instances of objects here are analogous to the initial of the added feature which are being adjusted areas of which pixel or area percentage to be augmented into the image [according to par. 0043]); and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images (Amon, par. [0046] discloses the synthetic augmented training images are used to train the machine learning model) and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask (Amon, par. [0047] discloses the augmented instances of object are used as annotations for supervised machine learning; wherein the augmenting can be used a inpainting technique to add the instances of the image into the starting image).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where the set of functions further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
Wherein having Muddala’s system having where the set of functions further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
However, Muddala in view of Sun and Amon does not explicitly teach the initial outline of the added feature as ground truth.
Dvornik teaches the initial outline of the added feature as ground truth (Section 4.6 discloses the objects being augmented into the image is also the ground truth for training the neural network model).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun and Amon of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Dvornik of having the initial outline of the added feature as ground truth.
Wherein having Muddala’s system of having where the initial outline of the added feature as ground truth.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve data augmentation for learning deep neural networks by leveraging image task-specific prior knowledge enabling improved object detection. Since both Muddala and Dvornik perform data generation for training of neural network. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Dvornik’s system improves data augmentation for learning deep neural networks by leveraging image task-specific prior knowledge enabling improved object detection, (see Dvornik’s Abstract).
Regarding claim 6, Muddala in view of Sun further in view of Amon and Dvornik teaches the method of claim 5, Muddala teaches further comprising: using the machine learning model (As discussed above in Muddala, the inpainting [hole-filling] such as illustrated in FIG. 3 to fille in the occluded region, example of the occluded image is shown in FIG. 5.d; the process of the filling is further disclosed in 2.4.3 wherein the filling happens so that only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects; the inpainting process here is to make sure that the foreground region is excluded in the process so that the foreground region is not to be having artifacts [not needed for filling of to be inpainted]) to predict an inpainted depth estimate of each augmented feature region (in another sense, the overlapped DDP is now corrected labeled and not needed to be inpainted further in the corrected patch as disclosed in 2.4.3; moreover, Amon further discloses [0010-0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image, including the image and the mask of the objects according to [0011]; therefore, the whole process of Muddala as discussed above in claim 11 to output a result include processing of the detection of the object and classify the object regions as foreground or background which is now taught by Amon to include using machine learning model); and refining each augmented feature region based at least in part on the inpainted depth estimate of each augmented feature region (As discussed in Muddala, as discussed above in claim 11, by using the input and the mask of FIG. 3 and Fig. 4 a refined mask is output including refined respective inpainting region [according to section 2.4.3 wherein the refined region including the generated threshold image]; moreover, Amon, par. [0047] discloses the augmented instances of object are used as annotations for supervised machine learning; wherein the augmenting can be used a inpainting technique to add the instances of the image into the starting image).
However, Muddala in view of Sun and Dvornik does not explicitly teach while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
Amon teaches while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image (Amon, [0039] discloses the training images used for training includes objects at different depth levels, wherein each objects have their map [each respective mask as claimed, here each respective mask has no antecedent basis hence, can be understood to be any map or mask of object respectively as taught in the art]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun and Dvornik of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
Wherein having Muddala’s method of having where while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
Regarding claim 13, Muddala in view of Sun and Amon teaches the method of claim 11.
However, Muddala in view of Sun does not explicitly teach further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
Amon teaches the set of functions further comprising: obtaining a plurality of training images (Amon, par. [0021] discloses the training include obtaining a number of training images being fed to the feature extractor); adding an image feature to each of the training images (Amon, par. [0021] discloses the feature extractor to extract features from the training images; furthermore, par. [0043] discloses the training images may include images that have same scene is combined or augmented with objects or objects of predetermined types, in other word, the object being extracted the features can be added into the training images to create augmented images to be part of the training images); creating a plurality of training masks corresponding to the plurality of training images (Amon, par. [0043] discloses generation of training images include augmenting the starting image with another instance of an object of a predetermined type, and for a plurality of images would include the use of a plurality of objects of predetermined types; moreover, these instances are represented by pixel map according to par. [0047] which is analogous to “mask”), wherein each training mask comprises an image feature region comprising an initial outline of the added feature (as disclosed in Amon’s par. [0043], the portion of the another instance of object can be understood as the image feature including an initial of the added feature as claimed; moreover, these objects are represented by object outline according to [0132]); augmenting each feature region by adjusting the initial outline of the added feature (Amon, par. [0043-0045] discloses the training images are generated synthetically by augmenting the instances of objects into the starting image, the instances of objects here are analogous to the initial of the added feature which are being adjusted areas of which pixel or area percentage to be augmented into the image [according to par. 0043]); and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images (Amon, par. [0046] discloses the synthetic augmented training images are used to train the machine learning model) and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask (Amon, par. [0047] discloses the augmented instances of object are used as annotations for supervised machine learning; wherein the augmenting can be used a inpainting technique to add the instances of the image into the starting image).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun and Dvornik of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
Wherein having Muddala’s method of having where further comprising: obtaining a plurality of training images; adding an image feature to each of the training images; creating a plurality of training masks corresponding to the plurality of training images, wherein each training mask comprises an image feature region comprising an initial outline of the added feature; augmenting each feature region by adjusting the initial outline of the added feature; and after augmenting each feature region of the plurality of masks, training the machine learning model using the plurality of training images and the plurality of masks using the initial outline of the added feature for inpainting each training image using a corresponding mask.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance (see Amon’s Par. [0005]).
However, Muddala in view of Sun and Amon does not explicitly teach the initial outline of the added feature as ground truth.
Dvornik teaches the initial outline of the added feature as ground truth (Section 4.6 discloses the objects being augmented into the image is also the ground truth for training the neural network model).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun and Amon of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Dvornik of having the initial outline of the added feature as ground truth.
Wherein having Muddala’s method of having where the initial outline of the added feature as ground truth.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve data augmentation for learning deep neural networks by leveraging image task-specific prior knowledge enabling improved object detection. Since both Muddala and Dvornik perform data generation for training of neural network. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Dvornik’s system improves data augmentation for learning deep neural networks by leveraging image task-specific prior knowledge enabling improved object detection, (see Dvornik’s Abstract).
Regarding claim 14, Muddala in view of Sun further in view of Amon and Dvornik teaches the method of claim 13, Muddala teaches further comprising: using the machine learning model to predict (As discussed above in Muddala, the inpainting [hole-filling] such as illustrated in FIG. 3 to fille in the occluded region, example of the occluded image is shown in FIG. 5.d; the process of the filling is further disclosed in 2.4.3 wherein the filling happens so that only background pixels in the target patch by removing foreground pixels in order to avoid artifacts at foreground objects; the inpainting process here is to make sure that the foreground region is excluded in the process so that the foreground region is not to be having artifacts [not needed for filling of to be inpainted]) an inpainted depth estimate of each augmented feature region (the overlapped DDP is now corrected labeled and not needed to be inpainted further in the corrected patch as disclosed in 2.4.3; moreover, Amon further discloses [0010-0011] discloses using machine learning model to the image to obtain background and foreground images classified for the image, including the image and the mask of the objects according to [0011]; therefore, the whole process of Muddala as discussed above in claim 11 to output a result include processing of the detection of the object and classify the object regions as foreground or background which is now taught by Amon to include using machine learning model); and refining each augmented feature region based at least in part on the inpainted depth estimate of each augmented feature region (As discussed in Muddala, by using the input and the mask of FIG. 3 and Fig. 4 a refined mask is output including refined respective inpainting region [according to section 2.4.3 wherein the refined region including the generated threshold image]; moreover, Amon, par. [0047] discloses the augmented instances of object are used as annotations for supervised machine learning; wherein the augmenting can be used a inpainting technique to add the instances of the image into the starting image).
However, Muddala in view of Sun and Dvornik does not explicitly teach while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
Amon teaches while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image (Amon, [0039] discloses the training images used for training includes objects at different depth levels, wherein each objects have their map [each respective mask as claimed, here each respective mask has no antecedent basis hence, can be understood to be any map or mask of object respectively as taught in the art]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun and Dvornik of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Amon of having where while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
Wherein having Muddala’s method of having where while training the machine learning model, applying each respective mask to a depth estimate of each corresponding training image.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance. Since both Muddala and Amon perform object localization using segmentation model based on classifying pixels. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Amon’s system improves object detection using neural network involving complexity information in the scene enabling real-time and reliable performance for video surveillance, (see Amon’s Par. [0005]).
Claims 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Suryanarayana M. Muddala et. al. (“Spatio-Temporal Consistent Depth-Image-Based Rendering using Layered Depth Image and Inpainting, Feb. 2016, EURASIP Journal on Image and Video Processing, Vol. 2016, Article Number 9” hereinafter as “Muddala”) in view of Xiaoyan Sun et. al. (“US 2008/0238942 A1” hereinafter as “Sun”) and Rohit R. Ranade et. al. (“US 2021/0125307 A1” hereinafter as “Ranade”).
Regarding claim 17, Muddala in view of Sun teaches the method of claim 7 (As shown in FIG. 3 and FIG. 4, the inpainted image has clear background and foreground determined).
However, Muddala in view of Sun does not explicitly teach defocusing inpainted portions of the inpainted image and surrounding areas of the inpainted image such that the subject remains in focus.
Ranade teaches defocusing inpainted portions of the inpainted image and surrounding areas of the inpainted image such that the subject remains in focus (Paragraphs [0050]-[0051] discloses, after the image is being inpainted [hole filled], apply shallow depth of field of the image based on the inpainting regions, here Ranade further teaches to apply the shallow depth of field to the inpainted image; Par. [0005] discloses “dolly zoom effect…the camera is at position A with…an object is under focus…the camera moves to position B while increasing its FoV to where position B is closer than position A to the object…After such a camera movement, the sizes of background objects and appear to shrink as more objects are brought into the image due to the increased FoV” wherein the object of desired target remains in focus while the objects of non-target defocused, wherein Muddala teaches the target object being object of inpainting and the others are not to be inpainted according to depth differences).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Ranade of having defocusing inpainted portions of the inpainted image and surrounding areas of the inpainted image such that the subject remains in focus.
Wherein having Muddala’s method of having defocusing inpainted portions of the inpainted image and surrounding areas of the inpainted image such that the subject remains in focus.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection by detecting background and foreground regions enabling effective adjusting of field of view and simulating of foreground and background objects. Since both Muddala and Ranade perform image inpainting using background-foreground regions. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Ranade’s system improves object detection by detecting background and foreground regions enabling effective adjusting of field of view and simulating of foreground and background objects, (see Ranade’s Par. [0005]).
Regarding claim 18, Muddala in view of Sun and Ranade teaches the method of claim 7.
However, Muddala in view of Sun does not explicitly teach further comprising: determining a number of image artifacts in the inpainted image, wherein each image artifact corresponds to an inpainted portion of the inpainted image; determining that the number of image artifacts exceeds a threshold number; and applying a shallow depth of field to the image based on the number of image artifacts exceeding the threshold number.
Ranade teaches further comprising: wherein applying the shallow depth of field to the inpainted image comprises: determining a number of image artifacts in the inpainted image (Ranade, par. [0009] discloses the final image [which is the final image result of the applying the shallow depth of field as disclosed in paragraphs [0050]-[0051] includes a window averaging filer being applied to the occluded areas in the fused image, which is further disclosed in par. [0126] which is using a kernel size to filter out occluded areas and par. [0125] if there is additional pixel locations, if there is additional pixel locations then perform the window averaging filter, therefore, there is a determination of number of additional pixel locations), wherein each image artifact corresponds to an inpainted portion of the inpainted image (Ranade, the occluded areas indicates the areas to be inpainted according to paragraphs [0050]-[0051]); determining that the number of image artifacts exceeds a threshold number (as discussed previously, as there is additional pixel locations, that the number of them exceeds one, then perform the window averaging filter according to Pars. [0050-0051]); and applying a shallow depth of field to the image based on the number of image artifacts exceeding the threshold number (the window averaging filter and the shallow depth of field being performed as a process to obtain the final image according to par. [0009]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Ranade of having further comprising: determining a number of image artifacts in the inpainted image, wherein each image artifact corresponds to an inpainted portion of the inpainted image; determining that the number of image artifacts exceeds a threshold number; and applying a shallow depth of field to the image based on the number of image artifacts exceeding the threshold number.
Wherein having Muddala’s method of having further comprising: determining a number of image artifacts in the inpainted image, wherein each image artifact corresponds to an inpainted portion of the inpainted image; determining that the number of image artifacts exceeds a threshold number; and applying a shallow depth of field to the image based on the number of image artifacts exceeding the threshold number.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve object detection by detecting background and foreground regions enabling effective adjusting of field of view and simulating of foreground and background objects. Since both Muddala and Ranade perform image inpainting using background-foreground regions. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Ranade’s system improves object detection by detecting background and foreground regions enabling effective adjusting of field of view and simulating of foreground and background objects, (see Ranade’s Par. [0005]).
Regarding claim 19, Muddala in view of Sun and Ranade teaches the method of claim 7, Muddala teaches further comprising: detecting one or more image artifacts corresponding to one or more inpainted portions of the inpainted image (Muddala, page 7, last par., discloses the background and the foreground regions have different depth estimates; the depth of the foreground is always less than the depth of the background as disclosed in page 6, 1st par. and illustrated in fig. 4; section 2.1.3 discloses the occlusion identification to correctly identify background and foreground and inpainting on the occluded areas [refining as claimed] including the mislabeled background and foreground points with overlapped labelled points in overlapped DDP as previously discussed in page 7, last par.; as the threshold images are created according to page 7, 2nd column, 1st par, done by adjusting the pixel of the occlusion depth by thresholding when it determines that the occlusion belongs to the background, hence, the thresholding is to omit the foreground feature processing, and that the background depth is determined to be background since its larger than the depth of foreground, as discussed previously); comparing, based on the depth representation, a depth of each image artifacts to a foreground depth of the inpainted imagen (as discussed previously, the occlusion mask is determined that the foreground and background depth of each image contains artifacts; as disclosed in Muddala’s page 2, 1st par., and figure 2a); and applying a shallow depth of field to the inpainted image based on determining that the depth of each image artifact is greater than the foreground depth (as the processing to obtain the inpainted image is taught in Ranade to apply the shallow depth of field).
Claims 22 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Suryanarayana M. Muddala et. al. (“Spatio-Temporal Consistent Depth-Image-Based Rendering using Layered Depth Image and Inpainting, Feb. 2016, EURASIP Journal on Image and Video Processing, Vol. 2016, Article Number 9” hereinafter as “Muddala”) in view of Xiaoyan Sun et. al. (“US 2008/0238942 A1” hereinafter as “Sun”) and Xiaoyi Huang et. al. (“US 2021/0027539 A1” hereinafter as “Sun”).
Regarding claim 22, Muddala in view of Sun teaches the system of claim 1, Muddala teaches wherein identifying the distractor for removal from the image (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”).
However, Muddala in view of Sun does not explicitly teach identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
Huang teaches identifying the distractor for removal from the image is based on user input outlining the distractor in the image (Par. [0032] discloses “while the image is displayed on the display device…the user can tap on depicted chair to be removed, drag a shape…around the depicted chair, perform a circular gesture around the depicted chair to roughly outline it”; moreover, Par. Par. [0033] discloses “the mask engine generates an image mask from the specified region…the original image with hole created by the image mask is the data used for inpainting and interpolation”).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a system that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Huang of having identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
Wherein having Muddala’s system of having identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve image rendering to simulate an image of an environment taking into consideration of spectral quality and reflectance and user selection enabling an increased in simulation accuracy. Since both Muddala and Huang perform image inpainting using user selection of image region. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Huang’s system improves image rendering to simulate an image of an environment taking into consideration of spectral quality and reflectance and user selection enabling an increased in simulation accuracy, (see Huang’s Par. [0050]).
Regarding claim 24, Muddala in view of Sun teaches the method of claim 7, Muddala teaches wherein identifying the distractor for removal from the image (Section 2.7 discloses “these artifacts are removed by using hole-filling post-processing”).
However, Muddala in view of Sun does not explicitly teach identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
Huang teaches identifying the distractor for removal from the image is based on user input outlining the distractor in the image (Par. [0032] discloses “while the image is displayed on the display device…the user can tap on depicted chair to be removed, drag a shape…around the depicted chair, perform a circular gesture around the depicted chair to roughly outline it”; moreover, Par. Par. [0033] discloses “the mask engine generates an image mask from the specified region…the original image with hole created by the image mask is the data used for inpainting and interpolation”).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Muddala in view of Sun of having a method that performs receiving an image from an image capture device, wherein the image represents a scene that includes a subject and a distractor that is visible in the image; identifying the distractor for removal from the image by inpainting, with the teachings of Huang of having identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
Wherein having Muddala’s method of having identifying the distractor for removal from the image is based on user input outlining the distractor in the image.
The motivation behind the modification would have been to improve image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, and further to improve image rendering to simulate an image of an environment taking into consideration of spectral quality and reflectance and user selection enabling an increased in simulation accuracy. Since both Muddala and Huang perform image inpainting using user selection of image region. Wherein Muddala’s system improves image inpainting that focuses on classifying occlusions during the inpainting process enabling reduction of spatial-temporal inconsistencies in the image, (see Muddala’s Abstract) and Huang’s system improves image rendering to simulate an image of an environment taking into consideration of spectral quality and reflectance and user selection enabling an increased in simulation accuracy, (see Huang’s Par. [0050]).
Pertinent Prior Art(s)
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Huang, Sheng-Wei et. al., “US 2023/0274400 A1”, discloses methods for accurately and efficiently removing objects from digital images taken from a camera viewfinder stream. For example, the disclosed systems access digital images from a camera viewfinder stream in connection with an undesired moving object depicted in the digital images. The disclosed systems generate a temporal window of the digital images concatenated with binary masks indicating the undesired moving object in each digital image. The disclosed systems further utilizes a generator as part of a 3D to 2D generative adversarial neural network in connection with the temporal window to generate a target digital image with the region associated with the undesired moving object in-painted. In at least one embodiment, the disclosed systems provide the target digital image to a camera viewfinder display to show a user how a future digital photograph will look without the undesired moving object.
Helm, Patrick A. et. al., “US 11138768 B2”, discloses selected artifacts, which may be based on distortions or selected attenuation features, may be reduced or removed from a reconstructed image. Various artifacts may occur due to the presence of a metal object in a field of view. The metal object may be identified and removed from a data that is used to generate a reconstruction.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHUONG HAU CAI whose telephone number is (571)272-9424. The examiner can normally be reached M-F 8:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHUONG HAU CAI/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
