Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim 7 appears to be significantly similar to claim 1. Further lines 1-2 of claim 7 indicate its dependency on claim 1, thus requiring the full incorporation of claim 1. Additionally, the specification was not found to have any indication that the method performed was separate to the device’s operations. [0021] and [0028] further suggests that the method is executed by the playout device. As such, claim 7 is being interpreted as a dependent claim whose terms are referencing back to claim 1. The terms listed below are not referenced back to in further limitations, but it is understood that said terms introduced in claim 7 that are also present in claim 1 (shown below) are not meant to be new and different instances introduced in claim 7.
As a result of the high level of similarity of claims 7 and 1 (as well as the specification suggesting that it is the method executed by the device of claim 1), the limitations
“…a working memory…” in claim 7 line 4
“…an input video stream…” in claim 7 line 6
“…a corresponding number of images…” in claim 7 line 8
“…a center…” in claim 7 line 10
will be interpreted to read as:
“…the working memory…” in claim 7 line 4
“…the input video stream…” in claim 7 line 6
“…the corresponding number of images…” in claim 7 line 8
“…the center…” in claim 7 line 10
Claim Rejections - 35 USC § 112
5. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “smooth movement” in claims 1 and 7 is a relative term which renders the claim indefinite. The term “smooth movement” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. For instance, if the movement has a jittery motion, but this jitter is within a desired threshold, is that considered smooth? Is it to be considered a smooth movement based on the complexity of the path of movement from the virtual camera? Would a certain number of direction changes (like having more than two directional changes) be non-smooth? It is unclear what determines the smoothness of a movement as well as what degree of said smoothness is required to be considered a “smooth movement”.
The term smooth will be omitted from the claim language, and the limitation “…a smooth movement…” will be read as “…a movement…” in claims 1 and 7, respectively.
Claim 1 recites the limitation "the associated zoom levels" in line 16. There is insufficient antecedent basis for this limitation in the claim. Due to lines 16-17, this is understood to not be in reference to the zoom levels of key points 1 to N (as the limitation states a basis on said zoom levels). Thus, it is unclear what zoom levels the limitation of line 16 is referring to.
The limitation "the associated zoom levels" in line 16 will be read as "the associated zoom levels of the virtual camera view” which is a set of zoom levels the is based on the zoom levels of the key points 1 to N per lines 13-15 of claim 1.
Claim 1 recites the limitation "…wherein the processor is configured to crop every image of the video clip outside the virtual camera view between the beginning and the end of the video clip to create a zoomed virtual camera clip." It is unclear in what manner the cropping is performed.
The cropping performed is stated to be for every image outside the virtual camera view point, but the claims suggest that every image frame in the defined image sequence has a virtual camera view point (per lines 13-15). As such, it is not clear what images are being cropped.
Additionally, the cropping is suggested to be done on the image data within the virtual camera view in light of the original disclosure, as suggested by Fig. 1 of the instant application. The limitations of the last 3 lines of claim 1 can be interpreted to mean cropping only for the data outside the virtual camera view (the action of “cropping” removes the outer edges of the image, thus cropping outside the virtual camera view would suggest removing the boarding image data of the area outside the virtual camera view). However, it is understood that the limitations of claim 1 are meant to indicate the image data outside the virtual camera view is to be removed by cropping out said image data surrounding the virtual camera view, as is done in Fig. 1A-1B.
The last 3 lines of claim 1 will be read as "…wherein the processor is configured to crop out image data of the video clip outside the virtual camera view for every image frame between the beginning and the end of the video clip to create a zoomed virtual camera clip."
Claim 7 recites similar limitations to that of claim 1. As such, it introduces a plurality of terms that were previously introduced by independent claim 1, while further referencing back to these re-introduced terms without clarifying which instance they are referencing. These terms render the claims indefinite due to it being unclear as to which instance is being referenced to. This applies to:
“…a video stream…”
“…a beginning and end…“
“…a video clip…”
“…a zoom level associated with the key points 1 to N…”
“…a trajectory…”
“an associated zoom level…”
“a zoomed virtual camera view…”
Each instance of these terms is referenced back to in claim 7, and as a result, renders the respective terms indefinite (as the same terms are provided in claim 1). These respective terms will be read with the word “the” in place of the word “a”/”an” with regards to claim 7.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-9 and 11-13 as best understood are rejected under 35 U.S.C. 102 (a)(1)/(a)(2) as being anticipated by Mao et al. (US 20240028904 A1; hereinafter “Mao”).
In re to claim 1, Mao teaches wherein: a playout device (image capture and processing system; Fig. 1 shows the system that performs the image processing of the device that executes the method of the system, as is corroborated by [0084]. See also Fig. 8B, which is a system that further includes cropping and scaling, as is described in [0039] and shown to be compatible with the system of Fig. 1 per [0160]) for a broadcast production system ([0097] discloses that the image capture and processing system may be a part of a smartphone, the system when comprising said smartphone being understood as a broadcast production system) comprising a storage device for storing video streams ([0105] discloses the use of a video storage device and video archive for stored video), wherein the playout device comprises a user interface for receiving instructions from a user ([0075] lines 15-24 discloses the inclusion of a user interface allowing for user input (understood as instructions from the user)), and a processor executing software ([0161] discloses that the system of Fig. 8A (and as such 8B per [0039]-[0040] which relates the figures) may implement its software using one or more processors), which implements a plurality of functionalities of the playout device enabling a user to ([0098] discloses the use of software in order to implement the methodology of the processing system): select a video stream stored on the storage device for being downloaded into a working memory of the processor ([0105] discloses that video frames may be drawn from a video source (including that of an archive containing stored video). Thus, disclosing the selection of a stored video stream on the storage device, correspondent to the claims. Additionally, as this is stored data that is further shown to be provided to additional engines (as shown in Fig. 8B), it is understood to additionally teach the downloading of video stream data to the working memory of the processor) and being displayed on a display device (Fig. 8B (814) and [0185] show the output of processed frame data as video. See also [0163] and Fig. 8A, which indicate outputs from the frame cropping and scaling system may be output to the display device of Fig. 8A (819));
define a beginning (IN) and an end (OUT) of a video clip in the downloaded video stream ([0171] discloses the input of a sequence of frames including an initial frame (understood as a beginning). Additionally, it is further stated that the sequence may be a video , and thus is understood to have a defined end point (see also [0172], which indicates the video may be previously recorded for playback, and as, such finite)); and
select key points 1 to N and a zoom level associated with the key points 1 to N in a corresponding number of images between the beginning and the end of the video clip (Fig. 13B discloses automatic zooming operations, which are performed in Figs. 8A-C per [0202] with respect to the determination of a scaling ratio and center point for a given frame (see also [0229], which denotes that center point determination is done for the initial and subsequent frames). It is understood that the key points are the various center points. It further understood that the diagonal length is indicative of an associated zoom level of said key points due to defining the scaling ratios as suggested by [0158]), wherein the key points 1 to N and the associated zoom level are manually selected or an algorithm is launched to automatically determine the key points 1 to N and the associated zoom level ([0189] discloses the determination of bounding box by automatic selection of a region of interest within an image);
wherein the processor is configured to compute a trajectory composed of points defining a center and an associated zoom level of a zoomed virtual camera view for every image frame ([0192] discloses processing of each frame for the center point and diagonal length) between the beginning and the end of the video clip ([0158] discloses the re-sizing and cropping of each frame. Further, [0192] as well as Fig. 13B disclose the use of each center point to determine object movement trajectory. Additionally, as shown in Fig. 13B and described in [0233], the trajectory is used to perform a smoothing operation that subsequently defines center values and associated zoom levels (being the center values and scaling ratios that have had a smoothing operation performed on them as a part of the re-sizing and cropping operations performed in Fig. 8B) of a zoomed virtual camera view (understood as the updated zooming region of Fig. 13B (1330))), wherein the computation of the trajectory and the associated zoom levels is based on the center and the associated zoom level of the key points 1 to N ([0192] discloses that the trajectory is determined using the center points and diagonal length of bounding boxes of each frame. Thus, it is understood to be computed using the center and associated zoom level of the key points 1 to N (each correspondent to the claims, respectively). Additionally, [0233] discloses the use of the diagonal lengths due to smoothing of scaling ratios to determine the scaling ratio of the updated zooming region) and performed such that a smooth movement of the virtual camera view between the key points 1 to N is achieved (Fig. 13B and [0158] discloses the smoothing of the movement trajectory for the updated zooming region based on the key points 1 to N (correspondent to the claims), and as such discloses movement of the virtual camera view between key points 1 to N), and wherein the processor is configured to crop every image of the video clip outside the virtual camera view between the beginning and the end of the video clip to create a zoomed virtual camera clip (Fig. 8B shows the input of the sequence of frames undergoing cropping and scaling to output a collection of frames. Further, [0158] discloses the re-sizing and cropping of each frame, the output of each from following these operations being understood to collectively be the zoomed virtual camera clip).
In re to claim 2 [dependent on claim 1]. Mao teaches wherein: the processor comprises one or several neural network(s) configured to determine the key points 1-N and/or the associated zoom levels in the video clip ([0251] discloses determination of bounding box center points using a convolutional neural network, as shown in Fig. 6A).
In re to claim 3 [dependent on claim 2], Mao teaches wherein: at least one of the neural networks is configured to determine regions of interest in the video clip based on saliency detection or object tracking, and wherein the regions of interest are smaller than images of the video clip ([0099] discloses the use of neural network based classification in order to determine bounding boxes (understood as regions of interest) using object detection and tracking results).
In re to claim 4 [dependent on claim 3], Mao teaches wherein: the processor is configured to determine the key points 1 to N and the associated zoom levels utilizing the determined regions of interest ([0158] discloses the determination of the key points and zoom levels (correspondent to the claims) according to bounding boxes).
In re to claim 5 [dependent on claim 1], Mao teaches wherein: the processor is configured to upscale an image resolution of the cropped image to the original image resolution of the downloaded video stream or to a user selectable image resolution (Fig. 16 shows the zooming process resulting in the output of image data of the same resolution as the original image. Thus, showing the upscaling of image resolution of the cropped image to the original image resolution).
In re to claim 6 [dependent on claim 1], Mao teaches wherein: the processor is configured to apply image improvement algorithms (Fig. 16 shows the zooming process resulting in the output of image data of the same resolution as the original image. Thus, showing the upscaling of image resolution of the cropped image to the original image resolution. It is understood that the upscaling operation is indicative of an image improvement algorithm).
As to claim 7, it recites the same limitations of claim 1 with the only significant inclusion being that it is the method performed by the device of claim 1. As such, it is rejected for the same reasons as provided for claim 1.
In re to claim 8 [dependent on claim 7], Mao teaches wherein: further comprising upscaling the resolution of the cropped images to the original image resolution of the downloaded video stream or to a user selectable image resolution (Fig. 16 shows the zooming process resulting in the output of image data of the same resolution as the original image. Thus, showing the upscaling of image resolution of the cropped image to the original image resolution, understood to be indicative of the resolution of the downloaded video stream due to the zooming process being performed by engines that received the sequence of frames, as shown in Fig. 8B).
In re to claim 9 [dependent on claim 8], Mao teaches wherein: further comprising applying image improvement algorithms on the upscaled images (Fig. 9B (944) shows that after scaling cropped image data, video stabilization is performed (understood to be an improvement algorithm performed on upscaled images). See also [0186], which shows the processes of Fig. 9A-9B may be performed by the system of Fig. 8B).
In re to claim 11 [dependent on claim 7], Mao teaches wherein: further comprising automatically determining regions of interest in the video clip, wherein the regions of interest are smaller than images of the downloaded video stream ([0189] discloses the determination of bounding box by automatic selection of a region of interest within an image. Further, the region is stated in [0189] to be a portion of the frame (thus being smaller than images of the downloaded video stream, correspondent to the claims)).
In re to claim 12 [dependent on claim 11], Mao teaches wherein: further comprising performing saliency detection or object tracking for detecting the regions of interest in the video clip ([0099] discloses the use of neural network based classification in order to determine bounding boxes (understood as regions of interest) using object detection and tracking results).
In re to claim 13 [dependent on claim 11], Mao teaches wherein: further comprising utilizing the detected regions of interest to determine the key points and the associated zoom levels ([0192] discloses the use of bounding boxes to determine a center point and the diagonal length of said bounding boxes. Thus, utilizing the detected regions of interest to determine key points and associated zoom levels (correspondent to the claims, respectively)).
In re to claim 14 [dependent on claim 1], Mao teaches wherein: a broadcast system comprising a playout device according to claim 1 (Fig. 1 shows the system that performs the image processing of the device that executes the method of the system, as is corroborated by [0084]. Additionally, [0097] discloses that the image capture and processing system may be a part of a smartphone. The smartphone that used being understood as the broadcast system).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Mao in further view of Ramaswamy (WO publication 2018152437 A1; hereinafter “Ramaswamy”).
In re to claim 10 [dependent on claim 7], Mao does not explicitly teach wherein: further comprising displaying the downloaded video stream on a display device.
However, in a similar field of endeavor, Ramaswamy teaches wherein: further comprising displaying the downloaded video stream on a display device ([0042] discloses the display of the original video stream concurrently with the zoomed stream).
Ramaswamy, like Mao, teaches a system that displays a zoomed video.
Therefore, , it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mao, to display the original video data, as taught by Ramaswamy. The motivation for the proposed modification would have been to allow a user to view the full video stream without needing to use a second display, as is performed in Ramaswamy [0042].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN M COOMBER whose telephone number is (571)270-0950. The examiner can normally be reached Monday - Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN M COOMBER/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights