DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/11/2025, 01/15/2026 and 07/16/2026 is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2 and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS (No. US-20150235408-A1 “Gross”) in view of LUVIZON (No. US-20220207663-A1 “Luvizon”).
Regarding claim 1, Gross teaches “An electronic device comprising:” (the programmable device 200; Para 0027);
“a display;” (a two-dimensional (2D) display; Para 0001);
“memory comprising one or more storage media storing instructions; and” (a memory 212, .... The programmable control unit(s) 216 may retrieve instructions from the memory 212 and the storage device 214; Para 0027);
“at least one processor comprising processing circuitry,
wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:” (which may be one or more central processing units (CPUs) and/or graphics processing units (GPUs), ... The programmable control unit(s) 216 may retrieve instructions from the memory 212 and the storage device 214 and execute the instructions using cache 218 to perform operations; Para 0027);
“display an image on the display;” (The parallax effect created by movement (or orientation change) of the observer in such an embodiment thus makes the image appear to be a 3D image, even though the actual display is a 2D display; Para 0025);
“based on displaying the image, receive an input to apply a three-dimensional visual effect to the image;” (The parallax simulation and the depth of field simulation can be controlled automatically based on available orientation data or programmatically by other user input, such as controls of a user interface (UI); Para 0020);
“based on the input, identify segmentation information indicating an object of the image;” (the input image can be segmented by depth; Para 0021); (segmentation may be accomplished using object recognition techniques that detect objects in the image and define segments associated with the recognized objects; Para 0039);
“identify the object from the segmentation information;” (Using a sequence of images that sweep through different focusing points a coarse model can be generated that separates a subject object from the background of the image; Para 0036); (object recognition techniques that detect objects in the image and define segments associated with the recognized objects; Para 0039);
“apply the three-dimensional visual effect by generating, on the display, a first video representing a background area within the image moved by a first distance beyond the object; and” (for rendering still images and videos that enable a conventional 2D display to give an observer the appearance that the 2D display is actually a window into a 3D world; Para 0015); (amount of occlusion determines the extent of the parallax effect that can be achieved, with the larger the amount of occlusion, the larger the extent of the parallax effect can be achieved; Para 0022);
“apply the three-dimensional visual effect by generating, on the display, a second video representing the background area within the image moved by a second distance beyond the object, the second distance being shorter than the first distance.” (for rendering still images and videos that enable a conventional 2D display to give an observer the appearance that the 2D display is actually a window into a 3D world; Para 0015); (amount of occlusion determines the extent of the parallax effect that can be achieved, with the larger the amount of occlusion, the larger the extent of the parallax effect can be achieved; Para 0022);
Gross discloses an electronic device with memory, processing units and a display. The display, displays a sourced image. UI input to apply to 3D effect, depth map and segmentation, as well as object/foreground identification and background a foreground layers.
While Gross fails to teach “based on the object including an edge of the image” and “based on the object being spaced apart from the edge of the image”.
Luvizon teaches “based on the object including an edge of the image:” (the extent of the parallax effect that can be performed for a given effect is limited by the amount of scale and motion that do not expose occluded regions... inpainting the image gaps is required; Para 0013);
“based on the object being spaced apart from the edge of the image:” (a cautiously pre-defined movement is not able to avoid exposing gaps in the multiplane layer; Para 0013); (can generate an adaptive multiplane image (AMI) representation, which also considers an inpainting method for filling occluded regions and a rendering method for automatically producing new views of the reconstructed 3D scene; Para 0014);
Luvizon discloses the scale and movement of segmented layers depending on the image content and whether the movement reveals gaps. It would be obvious to a person skilled in the art when implementing the parallax animation of Gross, to recognize the movement range should be adjusted according to the location and geometry of the foreground subject. It would be obvious to inspect the foreground object intersects the image boundary and select different movement for different object placements.
Gross and Luvizon are analogous art as both of them are related to image processing and image view.
The motivation for the above is to have accurate use of content aware parallax limits to reduce rendering artifacts.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross by based on the object including an edge of the image and based on the object being spaced apart from the edge of the image as taught by Luvizon.
Regarding claim 2, “The electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain an inpainting area that replaces the object by performing inpainting with respect to the object; and” (Portions of the background layers that were occluded by more foreground layers may become visible as the foreground layers are translated relative to the background layers. ... use image fill or other inpainting techniques to fill in areas of the background layers that were not visible in the original 2D image, .... insufficient to handle the view of the background through the foreground segment, and other fill techniques may be used; Para 0043); (inpaint holes in the more foreground segments to improve the 3D effect; Para 0049);
“obtain the background area including the inpainting area, and a remaining area of the image different from the object.” (Portions of the background layers that were occluded by more foreground layers may become visible as the foreground layers are translated relative to the background layers; Para 0043);
Gross discloses filling region with estimated background image information after the foreground object is segmented into separate layer, thus exposing the concealed object. Gross also discloses the background area from visible portions of the image outside the foreground object segments and the inpainting portion corresponding to the region that is hidden by the object. This teaches the subject matter of background area containing both the inpainting area and remaining area different from object.
Claim 11 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 1. Therefore, claim 11 limitations are also rejected with the same rationale as regarding claim 1.
Claim 12 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 2. Therefore, claim 12 limitations are also rejected with the same rationale as regarding claim 2.
Claim(s) 3-4 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON in further view of YANG (No. US-20150091900-A1 “Yang”).
Regarding claim 3, Gross further teaches “The electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
identify depth information corresponding to the image;” (a depth map is generated for the image, using any desired technique. ... the depth map is used to segment the image into 2 or more segments. ... the segments are rendered as a multilayer image; Para 0041);
While Gross and Luvizon fail to teach “wherein generating the second video comprises: based on identifying, from the depth information, at least a portion of the background area having a depth lower than a depth between the object and a camera capturing the image, generating the second video among the first video or the second video”.
Yang teaches “wherein generating the second video comprises:
based on identifying, from the depth information, at least a portion of the background area having a depth lower than a depth between the object and a camera capturing the image, generating the second video among the first video or the second video.” (the process of segmentation can leverage distance information contained within a depth map to separate foreground and background objects. In several embodiments, an object detection technique is used to identify the presence of a specific type of object (e.g. a face) ... An alpha matte αz can be determined by modeling and observed image Iz as a convex combination of a foreground image Fz and a background image Bz; Para 0042); (a distance and/or a range of distances corresponding to the distance from the viewpoint of the camera that captured the image to the detected face. The distance and/or range of distances can be utilized in the identification of pixels corresponding to the detected face and the background; Para 0044);
Yang discloses each image area is assigned a Z or depth position and the foreground, background and midground layers are formed according to the ranges of depth values. Thus, in combination teaching, obtaining depth information. Assigning depth values to image regions, segmenting an object and background using depth information, comparing the layers according to the depth and parallax movement according to scene geometry. While not explicitly stating generating a second video, it would be obvious the selecting of the short distance video based on depth inversion condition would be included.
Gross, Luvizon and Yang are analogous art as both of them are related to imaging processes and segmentation of images
The motivation for the above is to have accurate depth information of image for generation of video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross and Luvizon by generating the second video comprises: based on identifying, from the depth information, at least a portion of the background area having a depth lower than a depth between the object and a camera capturing the image, generating the second video among the first video or the second video as taught by Yang.
Regarding claim 4, Gross further teaches “The electronic device of claim 3, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
based on identifying, from the depth information, the object having a depth lower than a depth of the background area, generate the first video using the segmentation information.” (the image may be segmented into regions based on a depth ordering of the segments; Para 0032); (a depth map is generated for the image, using any desired technique. ... the depth map is used to segment the image into 2 or more segments. ... the segments are rendered as a multilayer image; Para 0041); (parallax effect can then be simulated by translating position of the segments relative to each other within the stack; Para 0021);
Gross discloses obtaining a depth map, identifying differently positioned foreground and background regions, segmenting the image based on depth ordering, arranging the foreground segment above the background and translating the segmented layer to generate parallax rendering.
Claim 13 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 3. Therefore, claim 13 limitations are also rejected with the same rationale as regarding claim 3.
Claim 14 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 4. Therefore, claim 14 limitations are also rejected with the same rationale as regarding claim 4.
Claim(s) 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON and in further view of YANG in further view of BILBREY (No. US-8619128-B2 “Bilbrey”).
Regarding claim 5, while Gross, Luvizon and Yang fail to teach the limitation of claim 5, Bilbrey teaches “The electronic device of claim 3, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
identify the depth information indicated by sensor data of a time-of-flight (ToF) sensor, or a light detection and ranging (LiDAR) sensor that is obtained together with the image.” (The 3-D sensor may be a depth or distance sensor that is configured to determine depth/distance of points within a scene. In preferred embodiments, the 3-D sensor may be lidar time-of-flight (TOF); Col 5, Line 11-14);
Bilbry supplies depth information though LiDAR ToF sensor operating as part of an imaging system. It would be obvious to person skilled in the art to utilize the ToF/LiDAR depth map of Bilbrey as an input of Gross because it provides direct scene distance measurements and avoids relying solely on estimated or manually created depth information.
Gross, Luvizon, Yang and Bilbrey are analogous art as both of them are related to image processing and capture, rendering and/or generating images.
The motivation for the above is to accurate depth information for efficient rendering of image.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross, Luvizon and Yang by identify the depth information indicated by sensor data of a time-of-flight (ToF) sensor, or a light detection and ranging (LiDAR) sensor that is obtained together with the image as taught by Bilbrey.
Claim 15 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 5. Therefore, claim 15 limitations are also rejected with the same rationale as regarding claim 5.
Claim(s) 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON and in further view of YANG in further view of TONG (No. US-10861176-B2 “Tong”).
Regarding claim 6, while Gross, Luvizon and Yang fail to teach the limitation of claim 6, Tong teaches “The electronic device of claim 3, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
identify the depth information corresponding to the image using a model trained to output depth values respectively corresponding to pixels of the image.” (In training the model to estimate the image depth, in a path (1), in FIG. 4, from the source view 405 of an input image is sent through a prediction model 410 to estimate the depth of each pixel for the depth image; Col 11, Line 8-11);
Tong discloses the input is an image, the system used the trained model to estimate the image depth and output the depth value for each pixel of the image. Thus, the result has depth information corresponding to the input image and the pixel depth values.
Gross, Luvizon, Yang and Tong are analogous art as both of them are related to depth image and image processing.
The motivation for the above is to have accurate depth information of image for efficient output of results.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross, Luvizon and Yang by identify the depth information corresponding to the image using a model trained to output depth values respectively corresponding to pixels of the image as taught by Tong.
Claim(s) 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON in further view of BRUNNER (No. US-8130226-B2 “Brunner”).
Regarding claim 7, Gross further teaches “wherein the information include:
a horizontal position of a first layer corresponding to the background area;” (background segment 110, mid-ground segment 120, and foreground segment 130 are stacked centered horizontally; Para 0025);
“a horizontal position of a second layer corresponding to the object; and” (the amount of translation to be applied monotonically increases with stack position such that the segments at the top of the stack (closest) receive the largest translation; Para 0023);
“a horizontal position of a virtual camera that is moved, to render the first video or the second video, within a virtual space including the first layer and the second layer.” (the observer perceives the effect of “looking behind” the closer image segments in the stack... cause a change in the relative position and orientation of the user and the device; Para 0023);
While Gross and Luvizon fail to teach “identify information indicating distinct three-dimensional visual effects, the information being defined with respect to key frames of a video which is a result of applying the three-dimensional visual effect”.
Brunner teaches “The electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
identify information indicating distinct three-dimensional visual effects, the information being defined with respect to key frames of a video which is a result of applying the three-dimensional visual effect,” (attributes describing the position of the affected layer in the layer tree will change; Col 4, Line 21-22); (performs explicit animation of the events or changes and configures the layout of the layers in the render tree; Col 5-6, Line 66-67, 1); (being moved from a start point A to an endpoint B (e.g., the application 110 may receive user input moving a view from a starting position on the display to an ending position); Col 2, Line 51-54);
Brunner discloses basic animation and keyframe animation of layers, like when the layer is moved, the animation system performs animation events and the initial and final states as skey states or key frames which the layer position is animated. Gross generates a parallax animation by placing an object and background on separated depth layers and changing their horizontal positions according to a changing viewpoint. Brunner teaches the representing the positions and state changes of the layers as animation attributes. It would be obvious to define the background layers position and foreground layer position and virtual cameras position at selected key frames because these basic variable at determine the parallax relationship at each point in the animation. Having these values at keyframes would produce the desired 3D visual effect.
Gross, Luvizon and Brunner are analogous art as both of them are related to rendering and visual effect.
The motivation for the above is to have accurate visual effects whit efficient key frames for video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross and Luvizon by identify information indicating distinct three-dimensional visual effects, the information being defined with respect to key frames of a video which is a result of applying the three-dimensional visual effect as taught by Brunner.
Claim 17 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 7. Therefore, claim 17 limitations are also rejected with the same rationale as regarding claim 7.
Claim(s) 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON and in further view of BRUNNER in further view of GEVER (No. US-6329994-B1 “Gever”).
Regarding claim 8, Gross further teaches “The electronic device of claim 7, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
based on identifying the object spaced apart from the edge of the image, change the horizontal position of the first layer indicated by the information such that a deviation of horizontal positions of the first layer across the key frames is decreased;” (a background view through a hole may be generated as a separate background segment or incorporated as part of a background segment, so that the background viewed through the hole changes when the foreground segment is translated to produce the parallax effect; Para 0045);
While Gross, Luvizon and Brunner fail to teach “generate a virtual space including the virtual camera, the second layer and the first layer which are sequentially positioned from the virtual camera”.
Gever teaches “generate a virtual space including the virtual camera, the second layer and the first layer which are sequentially positioned from the virtual camera; and generate the second video by moving, within the virtual space, the virtual camera, the second layer, and the first layer according to the information” (defining a three-dimensional geometrical description of an animation scene;.. rendering a first image of the scene as seen from a viewpoint at a first position on the geometrical path;.. translating the viewpoint to a second position along the geometrical path; and.. rendering a second image of the scene as seen from the second position; Col 10, Line 33-43); (renders animated images of a three-dimensional scene, each image rendered as Seen from a respective viewpoint, including a first image of the scene as seen from a first viewpoint along a predetermined geometrical path and a second image of the scene as seen from a Second Viewpoint; Col 12, Line 11-16);
“generate the second video by moving, within the virtual space, the virtual camera, the second layer, and the first layer according to the information.” (defining a three-dimensional geometrical description of an animation scene;.. rendering a first image of the scene as seen from a viewpoint at a first position on the geometrical path;.. translating the viewpoint to a second position along the geometrical path; and.. rendering a second image of the scene as seen from the second position; Col 10, Line 33-43);
Gever discloses an animation generator that renders images of 3D scene from a respective viewpoint and Gross supplies the use of separating foreground objects and background layers. A person of ordinary skill would use the keyframe virtual camera animation of Gever and have reduced the horizontal excursion of the background layer when the fully bounded foreground object is separated from the image edges. This would reduce the exposed gaps and maintain the parallax animation.
Gross, Luvizon, Brunner and Gever are analogous art as both of them are related to image processing and rendering.
The motivation for the above is to have a accurate rendering of virtual space to generate a parallax animation.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross, Luvizon and Brunner by generate a virtual space including the virtual camera, the second layer and the first layer which are sequentially positioned from the virtual camera and generate the second video by moving, within the virtual space, the virtual camera, the second layer, and the first layer according to the information as taught by Gever.
Claim 18 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 8. Therefore, claim 18 limitations are also rejected with the same rationale as regarding claim 8.
Claim(s) 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON in further view of PARK (No. US-10721391-B2 “Park”).
Regarding claim 9, while Gross and Luvizon fail to teach the limitation of claim 9, Park teaches “The electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain the segmentation information corresponding to the image by inputting the image into a model for object detection and receiving, as an output from the model, the segmentation information.” (the neural network of the mask branch (e.g., the fully convolutional network) would be configured to generate instance segmentation masks for object instances depicted in images; Col 37, Line 47-49); (a predicted classification and bounding box for each Rol. In certain circumstances; Col 37, Line 66-67); (generate masks for a subset of the Rols that are deemed likely to be of interest; Col 38 Line 3-4);
Gross, Luvizon and Park are analogous art as both of them are related to image processing and segmentation.
The motivation for the above is to have accurate segmentation information for efficient output from model.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross and Luvizon by obtain the segmentation information corresponding to the image by inputting the image into a model for object detection and receiving, as an output from the model, the segmentation information as taught by Park.
Claim 19 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 9. Therefore, claim 19 limitations are also rejected with the same rationale as regarding claim 9.
Claim(s) 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over GROSS in view of LUVIZON in further view of COHEN-MARTIN (No. US-20150221000-A1 “Cohen”).
Regarding claim 10, while Gross and Luvizon fail to teach the limitation of claim 10, Cohen teaches “The electronic device of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:
display, on the display, a visual object indicating to store the result, together with the result to which the three-dimensional visual effect is applied including the first video or the second video; and” (The Storyboard button causes the display of sequence of images 515 “summarizing” a video ad being created, for example. ... are the Save button 558 and the Save and Run button 559; Para 0201); (The “Save Button” 1030 updates the XML that was formed on the page; Para 0252);
“based on receiving an input with respect to the visual object, store the first video or the second video that is displayed on the display as the result.” (The “Save Button” 1030 updates the XML that was formed on the page; Para 0252);
Cohen discloses save button that stores result which showcases a visual object indicating storage. Although Cohen’s generated video isn’t a parallax video, Gross supplies the visual effect video which together results in the three-dimensional visual effect video. Gross generates the displayed 3D parallax video and Cohen provide the displaying the saved control and receiving selection of that control, storing the fata defining the video.
Gross, Luvizon and Cohen are analogous art as both of them are related to image/video processing.
The motivation for the above is to have efficient storage of video for storing results.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Gross and Luvizon by display, on the display, a visual object indicating to store the result, together with the result to which the three-dimensional visual effect is applied including the first video or the second video and based on receiving an input with respect to the visual object, store the first video or the second video that is displayed on the display as the result as taught by Cohen.
Claim 20 is directed to a method and its limitations are similar in scope and functions performed by the electronic device of claim 10. Therefore, claim 20 limitations are also rejected with the same rationale as regarding claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
CN-110933355-B (Van) – Discloses displaying visual effects in image data. In some examples, the visual effect includes an avatar displayed on the face of the user. In some examples, the visual effect includes a sticker applied to the image data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIGITER D PROTAZI whose telephone number is (571)272-7995. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A Broome can be reached at 5712722931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.D.P./Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
