Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 5, 6, 8-10, 12, 13, and 16-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by U.S. Patent Application Publication 2020/0035019 A1 (hereinafter Cappello).
Regarding claim 1, the limitations “a method comprising: receiving two-dimensional video content of a physical space captured by a camera system; generating a three-dimensional video segment having computer imagery corresponding to at least a portion of the physical space, including: computing movement data of an object in the two-dimensional video content; and generating a dynamic object for the object based on the movement data” are taught by Cappello (Cappello, e.g. abstract, paragraphs 24-101, discloses a system which receives one or more 2D video streams of a 3D scene including people and other objects as part of a live event such as a soccer game, and generates an interactive 3D virtual reconstruction of the live event, i.e. a 3D video segment. Cappello, e.g. paragraphs 24-45, describes camera placement and calculating 3D data from 2D video, wherein the 3D data includes tracking pose and motion data for people and objects using keypoint detection and tracking, e.g. paragraphs 46-51, 56, i.e. the claimed computed movement data of object(s) in the 2D video content. Further, Cappello, e.g. paragraphs 52-54, 58-71, teaches that the 3D virtual reconstruction includes models for the people and objects which are animated using the tracked pose and motion data, i.e. the claimed generated dynamic objects. Finally, it is noted with respect to claims 12 and 19, Capello, e.g. paragraph 101, teaches that the system may be implemented using a processor executing a program stored on a non-transitory medium.)
The limitation “rendering interactive video content for a display on a computing device based on the three-dimensional video segment, wherein movement of the dynamic object within the computer imagery displayed on the computing device corresponds to movement of the object in the two-dimensional video content” is taught by Cappello (Cappello, e.g. paragraphs 72-89, teaches that the 3D virtual reconstruction is provided to the view processor, which determines a virtual camera viewpoint based on user input, the contents of the input 2D video, and/or the content of the 3D scene, followed by rendering and displaying images of the 3D virtual reconstruction. It is noted that Cappello’s 3D virtual reconstructions are interactive in at least two ways, in that the user may control the virtual camera in addition to conventional video controls such as pause and play, e.g. paragraph 75, as well as by replacing one of the dynamic objects with a different model selected by the user, e.g. paragraph 71.)
Regarding claims 5 and 6, the limitations “further comprising: modifying the interactive video content based on a user selection to customize the interactive video content”, “wherein the user selection includes at least one of a viewing angle, an animation effect, or a characteristic of the dynamic object” are taught by Cappello (As discussed in the claim 1 rejection above, Cappello’s 3D virtual reconstructions are interactive in at least two ways, in that the user may control the virtual camera in addition to conventional video controls such as pause and play, e.g. paragraph 75, as well as by replacing one of the dynamic objects with a different model selected by the user, e.g. paragraph 71, i.e. the claimed user selection to customize the interactive video content by selecting a viewing angle, or a characteristic of the dynamic object.)
Regarding claim 8, the limitation “wherein generating the movement data includes estimating a three-dimensional position of a keypoint of the object from a two-dimensional position of the keypoint of the object identified in a plurality of frames of the two-dimensional video content” is taught by Cappello (As discussed in the claim 1 rejection above, Cappello, e.g. paragraphs 46-51, 56, describes tracking pose and motion data for people and objects using 3D keypoint detection and tracking. Cappello, e.g. paragraphs 46-51, indicates that the 2D video frames are analyzed to detect the locations of the players, the pose of each detected player, and the keypoints of the body, hands, and face of each detected player, which can be converted to 3D points in the 3D virtual reconstruction to determine the tracked keypoint positions over time, i.e. as claimed, 3D keypoint(s) of the object(s) are estimated from 2D keypoint(s) of the object(s) identified in the 2D video frames.)
Regarding claim 9, the limitations “wherein generating the dynamic object includes: generating a static object that represents the object by reconstructing a model from features extracted from the two-dimensional video content; and applying the movement data to the static object to transform the static object to the dynamic object” are taught by Cappello (As discussed in the claim 1 rejection, Cappello, e.g. paragraphs 52-54, 58-71, teaches that the 3D virtual reconstruction includes models for the people and objects which are animated using the tracked pose and motion data. Further, Cappello, e.g. paragraphs 63, 69, teaches that player model(s) may be generated using images from multiple cameras, known photogrammetry technique(s), and body mass estimation, i.e. as claimed, a static model is generated using features extracted from the two-dimensional video content, and the movement data is applied thereto to transform the static model to the dynamic object.)
Regarding claim 10, the limitations “generating identification data about the object in the two-dimensional video content; selecting, from an object model database, a static object based on the identification data; and applying the movement data to the static object to transform the static object to the dynamic object” are taught by Cappello (As discussed in the claim 1 rejection, Cappello, e.g. paragraphs 52-54, 58-71, teaches that the 3D virtual reconstruction includes models for the people and objects which are animated using the tracked pose and motion data. Further, Cappello, e.g. paragraphs 64-68, 70, teaches that player model(s) for identified players may be pre-defined and selected based on detection in the 2D video images, i.e. as claimed, a static model is selected from an object model database based on identification data about the object in the 2D video content, and the movement data is applied thereto to transform the static model to the dynamic object.)
Regarding claims 12 and 19, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 1 above.
Regarding claim 13, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claims 5 and 6 above.
Regarding claim 16, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 8 above.
Regarding claim 17, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 9 above.
Regarding claim 18, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 10 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2020/0035019 A1 (hereinafter Cappello) as applied to claim 1 above, and further in view of U.S. Patent Application Publication 2020/0169713 A1 (hereinafter Umemura).
Regarding claim 2, the limitations “receiving a user selection to view at least a portion of the two-dimensional video content in a three-dimensional format; and in response to the user selection, rendering the interactive video content for display on the computing device” are implicitly taught by Cappello (Cappello, e.g. paragraph 89, describes an embodiment wherein the 3D virtual reconstruction data is stored or exported, and later accessed by a video game type application for presenting the 3D virtual reconstruction to a user with additional 3D scene content generated by the application and/or interpolated from the record event data. Cappello does not explicitly describe how the interface works with respect to the user selecting a particular previously stored 3D virtual reconstruction file for access and interactive display, although one of ordinary skill in the art would have found it implicit that this embodiment would include a conventional file selection interface for allowing a user to select from a plurality of previously stored 3D virtual reconstruction files, i.e. providing a user the ability to store multiple files implicitly, if not inherently, requires providing the user the ability to later select from said multiple stored files. In the interest of compact prosecution, Umemura is cited for teaching an analogous system which generates interactive 3D virtual reconstructions from 2D video, stores the 3D virtual reconstructions in a storage unit, and later selects one or more of the stored 3D virtual reconstructions for access and interactive display as an augmented interactive 3D virtual reconstruction combining recorded events from the stored 3D virtual reconstruction(s).) However, this limitation is taught by Umemura (Umemura, e.g. abstract, paragraphs 19-123, discloses a system for generating interactive 3D virtual reconstructions from 2D video, e.g. paragraphs 20-23, 27-33, analogous to Cappello’s system. Umemura, e.g. paragraphs 28, 88-121, further describes storing the 3D virtual reconstructions in a storage unit, and allowing the user to select one or more of the stored 3D virtual reconstructions, e.g. paragraphs 42-50, 58-63, to access the contents thereof for generating an interactive 3D virtual reconstruction presenting the recorded event(s) with modifications or combined from multiple stored 3D virtual reconstructions, e.g. paragraphs 65-87. That is, as noted above, Umemura teaches what one of ordinary skill in the art would have found implicitly, if not inherently, required for Cappello’s video game application embodiment, i.e. providing a user interface for selecting at least one of the previously stored 3D virtual reconstruction file for access and interactive display.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Cappello’s video game application embodiment, in part, by providing a user interface for selecting previously stored 3D virtual reconstruction files for access and interactive display because one of ordinary skill in the art would have found it implicitly, if not inherently required, as noted above, and further in view of Umemura disclosing an analogous system providing an interface for a user to select from previously stored 3D virtual reconstruction files for generating modified interactive virtual viewpoint displays. In Cappello’s video game application embodiment providing a user interface for selecting previously stored 3D virtual reconstruction files for access and interactive display, as claimed, a user selection is received, and the corresponding 3D format representation of at least a portion of the 2D video content is rendered as interactive video content for display on the computing device.
Regarding claim 3, the limitation “in response to generating the three-dimensional video segment, storing the three-dimensional video segment in a memory device; and in response to the user selection: retrieving the three-dimensional video segment from the memory device; and rendering the interactive video content using the three-dimensional video segment” is taught by Cappello in view of Umemura (As discussed in the claim 2 rejection above, in Cappello’s video game application embodiment providing a user interface for selecting previously stored 3D virtual reconstruction files for access and interactive display, as claimed, a user selection is received, and the corresponding 3D format representation of at least a portion of the 2D video content is rendered as interactive video content for display on the computing device. Further, as implicitly required by Cappello’s video game application embodiment, and explicitly taught by Umemura, e.g. paragraphs 28, 88-121, prior to being selected for access and interactive display via the user interface, the 2D video segments are processed into the 3D virtual reconstruction files and saved to the storage unit, i.e. as claimed, the 3D video segment is stored and retrieved in response to the user selection for rendering the interactive video content.)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2020/0035019 A1 (hereinafter Cappello) as applied to claim 1 above, and further in view of U.S Patent Application Publication 2015/0131845 A1 (hereinafter Forouhar) in view of U.S Patent Application Publication 2019/0068945 A1 (hereinafter Maruyama).
Regarding claim 4, the limitations “detecting that a portion of the two-dimensional video content includes a key event; identifying a two-dimensional video segment from the portion of the two-dimensional video content; and generating the three-dimensional video segment from the two-dimensional video segment” are partially taught by Cappello (Cappello, e.g. paragraphs 76-80, teaches detecting key events in the 2D video content, but uses the event detection results to set a virtual camera angle for rendering the 3D video segment of the event from the portion of the 2D video content rather than for identifying a 2D video segment comprising the event for generating the 3D video segment. That is, Cappello’s event detection is not used for generating a separate 3D video segment/clip comprising the detected event using the corresponding 2D video segment/clip.) However, this limitation is taught by Forouhar in view of Maruyama (Forouhar, e.g. abstract, paragraphs 37-186, discloses a system for sports analytics which performs event detection using 4D tracking data derived from a plurality of 2D video cameras, e.g. paragraphs 37-41, 43, 45-49, 58-66. Further, Forouhar, e.g. paragraphs 77, 78, 92, 113, 119, uses the result of event detection to define highlight clip segments for review from the 2D video content, i.e. the claimed identifying 2D video segment(s) including detected key events. Finally, while Forouhar, e.g. paragraphs 93, 110, suggests generating a 3D recreation of a game from the 2D video content, analogous to Cappello’s system, Forouhar does not explicitly teach generating the 3D recreations for the event defined highlight clips as opposed to the entire game/activity. Maruyama, e.g. abstract, paragraphs 20-95, describes a system for generating interactive virtual viewpoint images from video captured from a plurality of viewpoints, e.g. paragraphs 20-22, analogous to Cappello’s system. Further, Maruyama, e.g. paragraphs 29, 30, 43-46, 51, 55, figures 2C, 4A, teaches providing a set of virtual viewpoint videos corresponding to different key events for selection by the user, i.e. generating 3D video segments from 2D video segments identified as including detected key events.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cappello’s system to include Maruyama’s detected event scene selection interface, wherein the scenes are defined using Forouhar’s highlight clips definition technique based on event detection results in order to provide the benefits of event based scene selection for 3D rendering from a recorded live event, as taught by Maruyama, for the different applications of sports analytics, as taught by Forouhar, paragraphs 77, 78, 92, 113, 119. In Cappello’s modified system, the results of event detection would be used to define 2D video highlight clip segments corresponding to detected events as taught by Forouhar, which would be available for user selection for 3D rendering as the claimed 3D video segment, as taught by Maruyama.
Claims 7, 11, 14, 15, 20, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2020/0035019 A1 (hereinafter Cappello) as applied to claims 1 12, and 19 above, and further in view of U.S Patent Application Publication 2019/0068945 A1 (hereinafter Maruyama).
Regarding claim 7, the limitation “wherein rendering the interactive video content includes: generating the interactive video content by executing a graphics rendering operation on the three-dimensional video segment, the graphic rendering operation including at least one of geometry processing, shading calculation, viewpoint calculation, or frame interpolation” is taught by Cappello (As noted in the claim 1 rejection, Cappello, e.g. paragraphs 72-89, teaches that the 3D virtual reconstruction is provided to the view processor, which determines a virtual camera viewpoint based on user input, the contents of the input 2D video, and/or the content of the 3D scene, followed by rendering and displaying images of the 3D virtual reconstruction. That is, Cappello’s graphics rendering includes viewpoint calculation of the virtual camera, and one of ordinary skill in the art would understand that animating the player representations or replacement avatars as in paragraphs 69-71 would include geometry processing, i.e. applying motion data to models for animation, and shading calculation, i.e. determining colors/shades of rendered pixels corresponding to the surfaces of the animated models.)
The limitation “transmitting, over a network, the interactive video content for display on the computing device without requiring the computing device to execute the graphics rendering operation” is not taught by Cappello (While Cappello, e.g. paragraph 52, indicates that the components of the system do not all need to be processed by the same computing device, Cappello does not explicitly teach that a first device performs part of the rendering, i.e. the claimed graphics rendering operation, and transmits the results to a second computing device for display without performing said graphics rendering operation.) However, this limitation is taught by Maruyama (Maruyama, e.g. abstract, paragraphs 20-95, describes a system for generating interactive virtual viewpoint images from video captured from a plurality of viewpoints, e.g. paragraphs 20-22, analogous to Cappello’s system. Further, Maruyama, e.g. paragraphs 38-44, 75-82, teaches that the image processing device which performs the virtual viewpoint image rendering and transmission to the user terminal, wherein the user terminal only performs the functions of display of the received virtual viewpoint images and receiving and transmitting user input to the image processing device, and the communication between devices may be via a network interface as in paragraphs 25, 40. It is additionally noted that one of ordinary skill in the art would recognize the benefit of Maruyama’s separate image processing and user terminal devices, i.e. the benefits of remote processing/rendering with a thin user client device as well known, including improved processing capability for the remote processing and reduced processing and/or power capabilities required for the user device.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Cappello’s system using Maruyama’s separate image processing and user terminal devices because, in addition to being analogous systems, one of ordinary skill in the art would be aware of the benefits of remote processing/rendering with a thin user client device, including improved processing capability for the remote processing and reduced processing and/or power capabilities required for the user device.
Regarding claims 11, 15, and 21, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claims 5-7 above. As noted in the claim 7 rejection, Maruyama, e.g. figure 1, paragraph 22, teaches that the user terminal receives the user input which is provided to the image processing device via the network.
Regarding claims 14 and 20, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 7 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT BADER whose telephone number is (571)270-3335. The examiner can normally be reached 11-7 m-f.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT BADER/Primary Examiner, Art Unit 2611
Prosecution Insights