DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
2. The information disclosure statements (IDS) submitted on the following dates are in compliance with the provisions of 37 CFR 1.97 and are being considered by the Examiner: 12/06/2024; 02/05/2026.
Specification
3. The title of the invention is not descriptive. A new title is required that is clearly indicative of
the invention to which the claims are directed.
The following title is suggested:
IMAGE PROCESSING METHOD, APPARATUS, DEVICE AND STORAGE MEDIUM FOR ADDING VIRTUAL OBJECT TARGET RENDER SET PORTION OBTAIN MAP GRAPH OVERLAP.
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1-5, 7, 9-10 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al. (machine translation of CN-110889890-A cited by applicant, hereinafter "Xie") in view of Wang et al. (machine translation of CN-112102340-A cited by applicant, hereinafter "Wang"), further in view of Wilson et al., (“Wilson”) [US-2020/0074642-A1]
Regarding claim 1, Xie discloses an image processing method (Xie- Fig. 1 shows a schematic flow diagram of an image processing method), comprising:
acquiring a first depth map of a virtual object and a second depth map of a standard virtual model relating to the target object (Xie- ¶0063-0064, at least disclose 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed. The first image to be processed contains a virtual object to be processed, and the second image to be processed contains the first object; ¶0069-0070, at least disclose 102. Render the virtual object to be processed and the first object to display the positional relationship between the virtual object to be processed and the first object as determined by the first depth image and the second depth image, and obtain the rendered image […] As described in 101, if the pixel values in the first depth image are used as the depth values of the pixels in the first image to be processed, then the depth value of each pixel in the first image to be processed can be determined based on the first depth image. Similarly, if the pixel values in the second depth image are used as the depth values of the pixels in the second image to be processed, then the depth value of each pixel in the second image to be processed can be determined based on the second depth image);
and
rendering the virtual object to obtain a virtual object map (Xie- ¶0021, at least discloses determining a virtual object that has a mapping relationship with the second object as the virtual object to be processed [virtual object map]; ¶0066, at least discloses The first image to be processed contains virtual objects to be processed [virtual object map]. Virtual objects to be processed refer to three-dimensional models with three-dimensional data constructed in a virtual three-dimensional space using 3D production software; ¶0069, at least discloses 102. Render the virtual object [rendering the virtual object] to be processed and the first object to display the positional relationship between the virtual object to be processed and the first object as determined by the first depth image and the second depth image, and obtain the rendered image); and
superimposing the set portion map and the virtual object map to obtain a target image (Xie- ¶0008, at least discloses using the pixel values in the first depth image as the depth values of the first image to be processed and the pixel values in the second depth image as the depth values of the second image to be processed, the depth information of the first image to be processed and the second image to be processed can be determined. Then, rendering processing is performed based on the relative depth information between the first image to be processed and the second image to be processed to obtain a rendered image, so as to accurately display the positional relationship between the virtual object to be processed and the first object).
Xie does not does not explicitly disclose dividing a set portion of a target object to obtain an initial mask map; adjusting the initial mask map based on the first depth map and the second depth map to obtain a target mask map; rendering the set portion based on the target mask map to obtain a set portion map
However, Wang discloses
dividing a set portion of a target object to obtain an initial mask map (Wang- Fig. 4 shows a human face mask 400 as an initial mask map; Fig. 5 shows an occluder mask map 500 as an initial mask map; ¶0061, at least discloses Step 102: Perform portrait segmentation on the acquired image to be processed to obtain a portrait mask [an initial mask map]; ¶0066, at least discloses Step 104: Perform object segmentation on the image to be processed to obtain the occlusion mask image; ¶0109, at least discloses The acquired image to be processed is subjected to human figure segmentation [dividing] processing to obtain human figure mask image 400 [obtain an initial mask map] […] Figure 4 shows a schematic diagram of a human face mask in one embodiment. Figure 4 includes a human face mask 400, a human face outline region 410 [set portion], and a background region 420 [set portion] […] The image to be processed is segmented to obtain the occlusion mask image 500. Figure 5 shows a schematic diagram of the mask of the occluder in one embodiment. Figure 5 includes an occlusion mask 500, an occlusion outline region 510 [set portion] (the white part in Figure 5), and a background region 520 [set portion] (the black part in Figure 5));
rendering the set portion based on the target mask map to obtain a set portion map (Wang- Fig. 6 shows the outline region of the portrait 610 [the set portion] based on target mask image 600 [the target mask map] to obtain the map of the outline region of the portrait 610 and the occluder mask image 500 [a set portion map]; ¶0066, at least discloses Step 104: Perform object segmentation on the image to be processed to obtain the occlusion mask image; ¶0069, at least discloses Step 106: Based on the position of the occlusion mask in the image to be processed, the occlusion mask and the portrait mask are fused to obtain the target mask; ¶0109, at least discloses Figure 6 shows a schematic diagram of a target mask);
Wang further discloses
superimposing the set portion map and the virtual object map to obtain a target image (Wang- ¶0069, at least discloses Step 106: Based on the position of the occlusion mask in the image to be processed, the occlusion mask and the portrait mask are fused to obtain the target mask; ¶0071, at least discloses Specifically, the electronic device merges each pixel in the occlusion mask with each pixel in the portrait mask based on the position of the occlusion mask in the image to be processed, to obtain the target mask. The target mask image includes the fusion result of blending the outline regions of the occluded objects and the outline regions of the human figure; Fig. 6 and ¶0109, at least disclose Figure 6 includes the target mask 600 [target image], the fusion result 610 obtained by fusing [superimposing] the human figure outline region and the object outline region, and the background region 620).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie to incorporate the teachings of Wang, and apply the segmentation on the acquired image to be processed to obtain a portrait mask into the Xie’s teachings for dividing a set portion of a target object to obtain an initial mask map; rendering the set portion based on the target mask map to obtain a set portion map.
Doing so would providing improving the accuracy of portrait segmentation.
The prior art does not explicitly disclose adjusting the initial mask map based on the first depth map and the second depth map to obtain a target mask map.
However, Wilson discloses
adjusting the initial mask map based on the first depth map and the second depth map to obtain a target mask map (Wilson- ¶0007-0008, at least disclose The adjusted segmentation mask [adjusting the initial mask map] can be used to generate an output frame with a modified visual effect. For example, background pixels of the output frame can be identified using the segmentation mask, and can be modified using a variable set of parameters […] The method further includes determining a segmentation mask using the first frame of the plurality of frames. The method further includes adjusting foreground pixels of the segmentation mask for the last frame of the plurality of frames relative to corresponding foreground pixels in the first frame. The foreground pixels can be adjusted using the cumulative optical flow map between the first frame and the last frame of the plurality of frames; ¶0018, at least discloses The foreground pixels of the output frame correspond to the adjusted foreground pixels of the segmentation mask […] the background pixels and the foreground pixels of the output frame are modified; ¶0067, at least discloses a pixel I(x, y, t) in the segmentation frame M can move by a distance (Δx, Δy) in a next frame M+t taken after a certain time Δt; ¶0050-0051, at least disclose depth-mapping techniques can generate and use depth maps [the first depth map and the second depth map] to determine which portion of a frame is foreground. For instance, data from a wide angle lens and data from a telephoto lens can be used to generate a depth map. The depth can then be used to manipulate certain objects in the frame. For example, background objects can be artificially blurred depending on how far they are from an in-focus object of interest […] Machine learning can also be used to generate a segmentation mask that indicates which pixels in a frame are foreground pixels and which pixels are background pixels […] the segmentation mask can include a first value for pixels that belong to an object of interest and a second value for pixels that belong to the background; ¶0079-0080, at least disclose when adjusting an inference segmentation mask, an optical flow vector may indicate that a foreground pixel has left the frame or has entered the frame […] If an optical flow vector indicates that a new pixel has entered the frame, the pixel can be designated as a foreground pixel or a background pixel based on a centroid (or other point) determined for the object of interest. For example, a distance of each foreground pixel of the object from the centroid can be known, and can be compared against the distance of the new pixel that has entered the frame. If the distance of the new pixel is within a certain distance of the centroid, the pixel can be assigned a foreground pixel value (e.g., a value of 1) in the segmentation mask […] Adjusted segmentation masks can be used by the rendering engine 110 to generate output frames 112 (e.g., output frame 200) with a modified visual effect [a target mask map] […] the pixels from the output frame that correspond to segmentation pixels from the segmentation mask that have a foreground value (e.g., a value of 1) are identified as foreground pixels. In some cases, the foreground pixels can be modified using a visual effect).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie/Wang to incorporate the teachings of Wilson, and apply the adjusted segmentation mask into the Xie/Wang’s teachings for adjusting the initial mask map based on the first depth map and the second depth map to obtain a target mask map.
Doing so would provide accurately segmenting images into foreground and background portions in a timely manner.
Regarding claim 2, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the acquiring of the first depth map of the virtual object comprises:
tracking an object adding portion based on a set tracking algorithm to obtain position information of the object adding portion; wherein the object adding portion is a portion of the target object to which the virtual object is added (Xie- ¶0105-0108, at least disclose 901. Obtain a reference image containing at least two reference feature points. In this embodiment of the disclosure, the camera needs to acquire a reference image before acquiring the second image to be processed, and the reference image contains at least two reference feature points. 902. Determine at least two feature points to be registered in the second image to be processed. One possible implementation is that the positions of at least two feature points to be registered in the second image to be processed are determined by optical flow tracking of at least two reference feature points; the reference feature points correspond one-to-one with the feature points to be registered.);
adding the virtual object to the object adding portion based on the position information (Xie- ¶0016, at least discloses firstly, the feature points to be registered in the second image to be processed are determined, and the feature points to be registered correspond one-to-one with the reference feature points; ¶0124, at least discloses when the ratio between the number of target pixels and the total number of pixels in the image area covered by the second object is less than a third threshold, that is, when the condition that the depth information of the second object is less than the depth information of the virtual object to be processed is not met, the relative depth information between the first image to be processed and the second image to be processed can be determined by comparing the depth values of the same pixel pairs in the first image to be processed and the second image to be processed by using the pixel values of the first depth image as the depth value of the first image to be processed and the pixel values of the second depth image as the depth value of the second image to be processed.); and
acquiring depth information of the virtual object after adding to obtain the first depth map (Xie- ¶0016, at least discloses firstly, the feature points to be registered in the second image to be processed are determined, and the feature points to be registered correspond one-to-one with the reference feature points; ¶0124, at least discloses when the ratio between the number of target pixels and the total number of pixels in the image area covered by the second object is less than a third threshold, that is, when the condition that the depth information of the second object is less than the depth information of the virtual object to be processed is not met, the relative depth information between the first image to be processed and the second image to be processed can be determined by comparing the depth values of the same pixel pairs in the first image to be processed and the second image to be processed by using the pixel values of the first depth image as the depth value of the first image to be processed and the pixel values of the second depth image as the depth value of the second image to be processed --> suggests pixel values of a second depth image as depth values of a second to-be-processed image, by comparing depth values of pairs of pixel points whose positions are the same in the first to-be-processed image and the second to-be-processed image, relative depth information between the first to be processed image and the second to be processed image may be determined, wherein the relative depth information comprises a positional relationship between the virtual object to be processed and the first object and a positional relationship between the virtual object to be processed and the second object, resulting in a rendered image. And an object addition location is the location of the target object where the virtual object is added and the virtual object is added to the object addition location based on the location information).
Regarding claim 3, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the standard virtual model is a virtual model of a shape of the target object, or the standard virtual model is a virtual model associated with a shape of the target object, or the standard virtual model is a virtual model constructed based on the target object of a current frame (Xie- ¶0059, at least discloses a "virtual camera" is not actual hardware. The principle behind a "virtual camera" [standard virtual model] taking "photos" is similar to that of a real camera or webcam in the real world, except that the specific process is achieved through software simulation. Specifically, the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels. The "virtual camera" emits a ray at the position of each pixel. For example, if ray A emitted from pixel A makes contact with the virtual model, then pixel A is the imaging pixel of the virtual model. The distance between the contact point of ray A and the virtual model and the "virtual camera" is the depth value of pixel A. If ray B emitted from pixel B does not make contact with the virtual model, then pixel B is the imaging pixel of the "background layer". The "background layer" can be regarded as a virtual model that is infinitely far away from the "virtual camera", so the depth value of pixel B is an infinite value. In addition, like a real camera, a "virtual camera" can freely rotate its viewpoint. The depth values mentioned above refer to the distance between the surface of the 3D virtual scene or virtual model and the "virtual camera" when the "virtual camera" images the 3D virtual scene or virtual model from a certain viewpoint.).
Regarding claim 4, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the adjusting of the initial mask map based on the first depth map and the second depth map to obtain the target mask map (see Claim 1 rejection for detailed analysis) comprises:
acquiring a near plane depth value and a far plane depth value of a virtual camera (Xie- ¶0058-0059, at least disclose The depth buffer also associates each depth value in the depth information with each pixel in the imaging plane of the "virtual camera". Here, the depth value refers to the distance between the 3D virtual scene or virtual model and the "virtual camera". The smaller the depth value, the closer the 3D virtual scene or virtual model is to the "virtual camera" […] the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels […] the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels);
linear transforming the first depth map and the second depth map according to the near plane depth value and the far plane depth value, respectively (Xie- ¶0015-0016, at least disclose obtaining a rotation matrix and a translation vector based on the reference three-dimensional coordinates of the reference feature points in the camera coordinate system and the three-dimensional coordinates to be registered of the feature points in the camera coordinate system, as the pose information of the camera; the overlap between the intermediate three-dimensional coordinates obtained by transforming the reference three-dimensional coordinates through the rotation matrix and the three-dimensional coordinates to be registered is greater than or equal to a second threshold […] firstly, the feature points to be registered in the second image to be processed are determined, and the feature points to be registered correspond one-to-one with the reference feature points; then, based on the reference three-dimensional coordinates of the reference feature points in the camera coordinate system and the three-dimensional coordinates of the feature points to be registered in the camera coordinate system, the rotation matrix and translation vector [linear transforming] are obtained; ¶0058-0059, at least disclose The depth buffer also associates each depth value in the depth information with each pixel in the imaging plane of the "virtual camera". Here, the depth value refers to the distance between the 3D virtual scene or virtual model and the "virtual camera". The smaller the depth value, the closer the 3D virtual scene or virtual model is to the "virtual camera" […] the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels […] the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels); and
adjusting the initial mask map based on the first depth map and the second depth map after linear transforming to obtain the target mask map (Xie- ¶0015-0016, at least disclose obtaining a rotation matrix and a translation vector based on the reference three-dimensional coordinates of the reference feature points in the camera coordinate system and the three-dimensional coordinates to be registered of the feature points in the camera coordinate system, as the pose information of the camera; the overlap between the intermediate three-dimensional coordinates obtained by transforming the reference three-dimensional coordinates through the rotation matrix and the three-dimensional coordinates to be registered is greater than or equal to a second threshold […] firstly, the feature points to be registered in the second image to be processed are determined, and the feature points to be registered correspond one-to-one with the reference feature points; then, based on the reference three-dimensional coordinates of the reference feature points in the camera coordinate system and the three-dimensional coordinates of the feature points to be registered in the camera coordinate system, the rotation matrix and translation vector [linear transforming] are obtained;
Wilson- ¶0007-0008, at least disclose The adjusted segmentation mask [adjusting the initial mask map] can be used to generate an output frame with a modified visual effect. For example, background pixels of the output frame can be identified using the segmentation mask, and can be modified using a variable set of parameters […] The method further includes determining a segmentation mask using the first frame of the plurality of frames. The method further includes adjusting foreground pixels of the segmentation mask for the last frame of the plurality of frames relative to corresponding foreground pixels in the first frame. The foreground pixels can be adjusted using the cumulative optical flow map between the first frame and the last frame of the plurality of frames; ¶0018, at least discloses The foreground pixels of the output frame correspond to the adjusted foreground pixels of the segmentation mask […] the background pixels and the foreground pixels of the output frame are modified; ¶0067, at least discloses a pixel I(x, y, t) in the segmentation frame M can move by a distance (Δx, Δy) in a next frame M+t taken after a certain time Δt; ¶0050-0051, at least disclose depth-mapping techniques can generate and use depth maps [the first depth map and the second depth map] to determine which portion of a frame is foreground. For instance, data from a wide angle lens and data from a telephoto lens can be used to generate a depth map. The depth can then be used to manipulate certain objects in the frame. For example, background objects can be artificially blurred depending on how far they are from an in-focus object of interest […] Machine learning can also be used to generate a segmentation mask that indicates which pixels in a frame are foreground pixels and which pixels are background pixels […] the segmentation mask can include a first value for pixels that belong to an object of interest and a second value for pixels that belong to the background; ¶0079-0080, at least disclose when adjusting an inference segmentation mask, an optical flow vector may indicate that a foreground pixel has left the frame or has entered the frame […] If an optical flow vector indicates that a new pixel has entered the frame, the pixel can be designated as a foreground pixel or a background pixel based on a centroid (or other point) determined for the object of interest. For example, a distance of each foreground pixel of the object from the centroid can be known, and can be compared against the distance of the new pixel that has entered the frame. If the distance of the new pixel is within a certain distance of the centroid, the pixel can be assigned a foreground pixel value (e.g., a value of 1) in the segmentation mask […] Adjusted segmentation masks can be used by the rendering engine 110 to generate output frames 112 (e.g., output frame 200) with a modified visual effect [a target mask map] […] the pixels from the output frame that correspond to segmentation pixels from the segmentation mask that have a foreground value (e.g., a value of 1) are identified as foreground pixels. In some cases, the foreground pixels can be modified using a visual effect)..
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Xie/Wilson, and apply adjusting the initial mask map after linear transforming into the Xie/Wang’s teachings for linear transforming the first depth map and the second depth map according to the near plane depth value and the far plane depth value, respectively; and adjusting the initial mask map based on the first depth map and the second depth map after linear transforming to obtain the target mask map.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 5, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the adjusting of the initial mask map based on the first depth map and the second depth map to obtain the target mask map (see Claim 1 rejection for detailed analysis) comprises:
in response to a first depth value in the first depth map being greater than a second depth value in the second depth map, keeping a pixel value of a corresponding pixel in the initial mask map unchanged (Xie- Fig. 5 and ¶0071, at least disclose Figure A represents the first image to be processed, Figure B represents the second image to be processed, and Figure C represents the rendered image. The depth values of pixel pairs with the same position in the first image to be processed and the second image to be processed are compared. That is, the depth values of pixel A1 and pixel B1, pixel A2 and pixel B2, pixel A3 and pixel B3, and pixel A4 and pixel B4 are compared […] when the depth value of pixel A2 is greater than the depth value of pixel B2, pixel B2 is rendered; ¶0090, at least discloses Determine the pixels to be processed in the second image to be processed whose depth value is less than or equal to the first threshold. The depth value is determined based on the second depth image);
in response to the first depth value being less than or equal to the second depth value, adjusting a pixel value of a corresponding pixel in the initial mask map to a set value (Xie- ¶0024, at least discloses the rendering processing unit is specifically configured to: use pixels in the second image to be processed whose depth values determined according to the second depth image are greater than a first threshold as pixels to be updated; Fig. 5 and ¶0071, at least disclose Figure A represents the first image to be processed, Figure B represents the second image to be processed, and Figure C represents the rendered image. The depth values of pixel pairs with the same position in the first image to be processed and the second image to be processed are compared. That is, the depth values of pixel A1 and pixel B1, pixel A2 and pixel B2, pixel A3 and pixel B3, and pixel A4 and pixel B4 are compared […] When the depth value of pixel A1 is less than the depth value of pixel B1, pixel A1 is rendered, meaning it is used as the pixel at the corresponding position in the rendered image to be displayed on the screen).
Regarding claim 7, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the rendering of the set portion based on the target mask map (see Claim 1 rejection for detailed analysis) comprises:
fusing image information corresponding to the target mask map with image information corresponding to an original map of the set portion to obtain fused image information (Wang- Fig. 1 and ¶0060-0069, at least disclose Step 102: Perform portrait segmentation on the acquired image to be processed to obtain a portrait mask […] Step 104: Perform object segmentation on the image to be processed to obtain the occlusion mask image […] Step 106: Based on the position of the occlusion mask in the image to be processed, the occlusion mask and the portrait mask are fused to obtain the target mask); and
rendering the set portion based on the fused image information (Wang- Fig. 1 and ¶0060-0069, at least disclose Step 102: Perform portrait segmentation on the acquired image to be processed to obtain a portrait mask […] Step 104: Perform object segmentation on the image to be processed to obtain the occlusion mask image […] Step 106: Based on the position of the occlusion mask in the image to be processed, the occlusion mask and the portrait mask are fused to obtain the target mask; ¶0172, at least discloses The mask fusion module 906 is used to fuse the occlusion mask and the portrait mask based on the position of the occlusion mask in the image to be processed, so as to obtain the target mask).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie/Wilson to incorporate the teachings of Wang, and apply the segmentation process is performed on the acquired to-be-processed image to obtain a portrait mask map into the Xie/Wilson’s teachings for fusing image information corresponding to the target mask map with image information corresponding to an original map of the set portion to obtain fused image information; and rendering the set portion based on the fused image information.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 9, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein there are a plurality of virtual objects, and the acquiring of the first depth map of the virtual object (Xie- ¶0063, at least discloses 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed; Fig. 2 and ¶0066, at least discloses The first image to be processed contains virtual objects to be processed. Virtual objects to be processed refer to three-dimensional models with three-dimensional data constructed in a virtual three-dimensional space using 3D production software. For example, scenes, people, objects, etc. in 3D game screens are all virtual three-dimensional models, but they are displayed on a two-dimensional display screen […] as shown in Figure 2, which is a scene diagram of virtual objects, the dinosaur shown in Figure A and the deer shown in Figure B are both virtual objects.) comprises:
acquiring a plurality of first depth maps respectively corresponding to the plurality of virtual objects (Xie- ¶0063-0069, at least discloses 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed […] the pixel values in the first image to be processed and the second image to be processed represent color information, and the pixel values in the first depth image and the second depth image represent depth information, that is, the distance from the image acquisition device to each point in the scene. The first image to be processed and the first depth image are created by a "virtual camera" in the physics engine (the physics engine can be Unity3D as described above, or other physics engines, which are not limited in this disclosure). The second image to be processed and the second depth image are created by a real camera (the real camera here is relative to the "virtual camera" […] 102. Render the virtual object to be processed and the first object to display the positional relationship between the virtual object to be processed and the first object as determined by the first depth image and the second depth image, and obtain the rendered image);
wherein the rendering of the virtual object comprises:
rendering the plurality of virtual objects based on the plurality of first depth maps (Xie- ¶0063-0069, at least discloses 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed […] the pixel values in the first image to be processed and the second image to be processed represent color information, and the pixel values in the first depth image and the second depth image represent depth information, that is, the distance from the image acquisition device to each point in the scene. The first image to be processed and the first depth image are created by a "virtual camera" in the physics engine (the physics engine can be Unity3D as described above, or other physics engines, which are not limited in this disclosure). The second image to be processed and the second depth image are created by a real camera (the real camera here is relative to the "virtual camera" […] 102. Render the virtual object to be processed and the first object to display the positional relationship between the virtual object to be processed and the first object as determined by the first depth image and the second depth image, and obtain the rendered image).
Regarding claim 10, Xie in view of Wang and Wilson, discloses the method according to claim 1, and discloses the method further comprising:
in response to there being a plurality of virtual objects, determining a set virtual object from the plurality of virtual objects (Xie- ¶0063, at least discloses 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed; ¶0072, at least discloses by using the pixel values in the first depth image as the depth value of the first image to be processed and the pixel values in the second depth image as the depth value of the second image to be processed, the depth information of the first image to be processed and the second image to be processed can be determined. Then, rendering processing is performed based on the relative depth information between the first image to be processed and the second image to be processed to obtain a rendered image, so as to accurately display the positional relationship between the virtual object to be processed and the first object);
wherein the acquiring of the second depth map of the standard virtual model relating to the target object (Xie- ¶0063, at least disclose 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed. The first image to be processed contains a virtual object to be processed, and the second image to be processed contains the first object;) comprises:
acquiring depth information of the set virtual object and the standard virtual model using a virtual camera to obtain the second depth map (Xie- ¶0059, at least discloses a "virtual camera" is not actual hardware. The principle behind a "virtual camera" [standard virtual model] taking "photos" is similar to that of a real camera or webcam in the real world, except that the specific process is achieved through software simulation. Specifically, the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane; ¶0063, at least discloses 101. Obtain the first image to be processed, the first depth image of the first image to be processed, the second image to be processed, and the second depth image of the second image to be processed; ¶0072, at least discloses by using the pixel values in the first depth image as the depth value of the first image to be processed and the pixel values in the second depth image as the depth value of the second image to be processed, the depth information of the first image to be processed and the second image to be processed can be determined. Then, rendering processing is performed based on the relative depth information between the first image to be processed and the second image to be processed to obtain a rendered image, so as to accurately display the positional relationship between the virtual object to be processed and the first object)).
Regarding claim 13, Xie in view of Wang and Wilson, discloses an electronic device (Xie- ¶0032, at least discloses an electronic device is provided), comprising:
at least one processor (Xie- ¶0032, at least discloses an electronic device is provided, comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, and the memory stores program instructions; when the program instructions are executed by the processor, the processor performs the method); and a memory, configured to store a program; when the program is executed by the at least one processor, the at least one processor (Xie- ¶0032, at least discloses an electronic device is provided, comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, and the memory stores program instructions; when the program instructions are executed by the processor, the processor performs the method) is caused to perform the method of claim 1.
Regarding claim 14, Xie in view of Wang and Wilson, discloses a computer readable medium, and further discloses wherein a computer program is stored on the computer readable medium, and when the computer program is executed by a processing apparatus (Xie- ¶0032, at least discloses an electronic device is provided, comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, and the memory stores program instructions; when the program instructions are executed by the processor, the processor performs the method), the computer program implements an image processing method comprising the method of claim 1.
6. Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Xie in view of Wang, further in view of Wilson, still further in view of Xu et al., (“Xu”) [US-2007/0036432-A1]
Regarding claim 6, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the adjusting of the initial mask map based on the first depth map and the second depth map to obtain the target mask map (see Claim 1 rejection for detailed analysis) comprises:
acquiring a two-dimensional map of the virtual object (Xie- ¶0059, at least discloses the process of a "virtual camera" taking a picture of a virtual model can be understood as a process of projecting a 3D model onto a 2D plane. The imaging plane of the "virtual camera" is composed of multiple pixels; ¶0091, at least discloses the 3D coordinates of these pixels in the camera coordinate system are obtained, that is, the coordinates of the projection points of the pixels in the 2D image onto the 3D space in the camera coordinate system);
in response to a first depth value in the first depth map being greater than a second depth value in the second depth map (see Claim 5 rejection for detailed analysis), keeping a pixel value of a corresponding pixel in the initial mask map unchanged (Xie- Fig. 5 and ¶0071, at least disclose Figure A represents the first image to be processed, Figure B represents the second values of pixel pairs with the same position in the first image to be processed and the second image to be processed are compared. That is, the depth values of pixel A1 and pixel B1, pixel A2 and pixel B2, pixel A3 and pixel B3, and pixel A4 and pixel B4 are compared […] when the depth value of pixel A2 is greater than the depth value of pixel B2, pixel B2 is rendered);
in response to the first depth value being less than or equal to the second depth value (see Claim 5 rejection for detailed analysis).
The prior art does not explicitly disclose in response to the first depth value being less than or equal to the second depth value, subtracting a set channel value of a corresponding pixel in the two-dimensional map from a pixel value of a corresponding pixel in the initial mask map to obtain a final pixel value.
However, Xu discloses
subtracting a set channel value of a corresponding pixel in the two-dimensional map from a pixel value of a corresponding pixel in the initial mask map to obtain a final pixel value (Xu- ¶0004, at least discloses A colour channel background subtraction technique is then performed, wherein for any particular input image, the RGB channels of the input image pixels are compared with the adaptive image to be processed, and Figure C represents the rendered image. The depth background, and dependent on the results of a logical comparison of the respective input and background R, G, or B values a pixel is set as either “foreground” or “background”. The map of “foreground” pixels constitutes a mask which is then used subsequently for further processing; ¶0007, at least discloses reference is made to the “mask” produced by the RGB subtraction method described earlier, in that each “background” pixel which is part of a hole in a connected object is set to foreground if the mask from the RGB subtraction indicates that it is foreground; ¶0028, at least discloses providing an improved pixel segmentation technique which accounts for “holes” left in the segmentation due to the shadow processing prior to performing a connected component analysis to identify connected objects […] This is achieved by applying a foreground segmentation technique as is known in the art to give a “mask” for the boundary of any segmented blobs (a “blob” being a group of adjacent segmented pixels), and applying shadow processing techniques similar to those described by McKenna et al. to identify shadows and highlights in the input image. As also mentioned by McKenna et al. the shadow processing results in “skeleton” segmented blobs being left, which blobs may contain holes, be reduced in size, or even bisected in comparison to the blobs obtained from the background subtraction. Before applying connected component analysis to the blobs to find connected objects a morphological dilation operation is applied to the skeleton blobs to dilate the pixels therein, the dilation operation being repeatedly applied so as to reconstruct the skeleton blobs until the re-constructed blobs touch the respective boundary of the mask of the corresponding blob (located at the substantially same position in the image as the blob or blobs being re-constructed) obtained from the background subtraction).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie/Wang/Wilson to incorporate the teachings of Xu, and apply the RGB subtraction into the Xie/Wang/Wilson’s teachings in response to the first depth value being less than or equal to the second depth value, subtracting a set channel value of a corresponding pixel in the two-dimensional map from a pixel value of a corresponding pixel in the initial mask map to obtain a final pixel value.
Doing so would provide an object detection method and system which is robust to illumination changes causing shadows and/or highlights.
7. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Xie in view of Wang, further in view of Wilson, still further in view of Xia et al., (“Xia”) [US-2021/0073982-A1]
Regarding claim 8, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses wherein the rendering of the set portion based on the target mask map (see Claim 1 rejection for detailed analysis) comprises:
rendering the set portion (see Claim 1 rejection for detailed analysis).
The prior art does not explicitly disclose determining transparency information of a pixel in an original map of the set portion according to the target mask map; wherein a pixel value of the pixel in the target mask map characterizes transparency; and rendering the set portion based on transparency information.
However, Xia discloses
determining transparency information of a pixel in an original map of the set portion according to the target mask map; wherein a pixel value of the pixel in the target mask map characterizes transparency (Xia- ¶0072, at least discloses the segmented image can be extracted from the image to be processed or the medical image by means of related calculation of the mask region and the medical image. For example, a transparent mask region is added onto an all-black image to obtain an image having a region to be transparent, after the image is overlapped with the corresponding image to be processed or the medical image, the segmented image only including the second target is generated. Or the all-black region is cut out from the overlapped image to obtain the segmented image. For another example, an all-white image adds a transparent mask region to obtain an image having a region to be transparent, after the image is overlapped with the corresponding medical image, the segmented image only including the second target is generated.); and
rendering the set portion based on transparency information (Xia- ¶0072, at least discloses the segmented image can be extracted from the image to be processed or the medical image by means of related calculation of the mask region and the medical image. For example, a transparent mask region is added onto an all-black image to obtain an image having a region to be transparent, after the image is overlapped with the corresponding image to be processed or the medical image, the segmented image only including the second target is generated. Or the all-black region is cut out from the overlapped image to obtain the segmented image. For another example, an all-white image adds a transparent mask region to obtain an image having a region to be transparent, after the image is overlapped with the corresponding medical image, the segmented image only including the second target is generated).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie/Wang/Wilson to incorporate the teachings of Xia, and apply the transparent mask region into the Xie/Wang/Wilson’s teachings for determining transparency information of a pixel in an original map of the set portion according to the target mask map; wherein a pixel value of the pixel in the target mask map characterizes transparency; and rendering the set portion based on transparency information.
Doing so would provide comprehensive, complete, and effective information to the medical personnel.
8. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Xie in view of Wang, further in view of Wilson, still further in view of Limonov, (“Limonov”) [US-2010/0104219-A1]
Regarding claim 11, Xie in view of Wang and Wilson, discloses the method according to claim 1, and further discloses after obtaining the target mask map, the method further comprising:
wherein the rendering of the set portion based on the target mask map to obtain the set portion map (see Claim 1 rejection for detailed analysis); and the rendering of the virtual object to obtain the virtual object map (see Claim 1 rejection for detailed analysis) comprise:
for a current frame, rendering the set portion based on a target mask map corresponding to a set forward frame to obtain the set portion map (Wilson- ¶0066, at least discloses The incremental dense optical flow maps can be computed between adjacent frames of the additional frames (e.g., between sets of adjacent frames f_n and f_{n−1}). Two adjacent frames can include two directly adjacent frames that are consecutively captured frames or two frames that are a certain distance apart (e.g., within two frames of one another, within three frames of one another, or other suitable distance) in a sequence of frames; Wang- Fig. 6 shows the outline region of the portrait 610 [the set portion] based on target mask image 600 [the target mask map] to obtain the map of the outline region of the portrait 610 and the occluder mask image 500 [a set portion map]; ¶0066, at least discloses Step 104: Perform object segmentation on the image to be processed to obtain the occlusion mask image; ¶0069, at least discloses Step 106: Based on the position of the occlusion mask in the image to be processed, the occlusion mask and the portrait mask are fused to obtain the target mask; ¶0109, at least discloses Figure 6 shows a schematic diagram of a target mask)); and
rendering a virtual object corresponding to the set forward frame to obtain the virtual object map (Xie- ¶0021, at least discloses determining a virtual object that has a mapping relationship with the second object as the virtual object to be processed [virtual object map]; ¶0066, at least discloses The first image to be processed contains virtual objects to be processed [virtual object map]. Virtual objects to be processed refer to three-dimensional models with three-dimensional data constructed in a virtual three-dimensional space using 3D production software; ¶0069, at least discloses 102. Render the virtual object [rendering the virtual object] to be processed and the first object to display the positional relationship between the virtual object to be processed and the first object as determined by the first depth image and the second depth image, and obtain the rendered image; Wilson- ¶0066, at least discloses The incremental dense optical flow maps can be computed between adjacent frames of the additional frames (e.g., between sets of adjacent frames f_n and f_{n−1}). Two adjacent frames can include two directly adjacent frames that are consecutively captured frames or two frames that are a certain distance apart (e.g., within two frames of one another, within three frames of one another, or other suitable distance) in a sequence of frames).
The prior art does not explicitly disclose caching the target mask map.
However, Limonov discloses
caching the target mask map (Limonov- Fig. 4 and ¶0056, at least disclose a mask buffer 430; ¶0062-0063, at least disclose The background depth information represents a depth value of a background included in a frame and the object depth information represents a depth value of an object included in the frame. The object is an individual object other than the background in the frame. Information on a mask may be defined as object region information for an object included in a currently displayed frame. In this case, the mask buffer 430 temporarily stores the mask to be applied to the frame. The mask can have a portion corresponding to the object that is in a color different from that of another portion of the mask or is perforated along the shape of the object […] The depth map generator 440 generates a depth map of the frame using the background depth information and the object depth information received from the meta data analyzer 420 and/or the mask received from the mask buffer 430).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Xie/Wang/Wilson to incorporate the teachings of Limonov, and apply the mask buffer into the Xie/Wang/Wilson’s teachings for caching the target mask map.
Doing so would minimize the size of a hole in a three-dimensional (3D) image generated from the 2D image when the 3D image is generated from the 2D image.
Conclusion
9. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. They are as recited in the attached PTO-892 form.
10. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL LE whose telephone number is (571)272-5330. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL LE/Primary Examiner, Art Unit 2614
Prosecution Insights