DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 11/18/2024 & 12/03/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because of the use of foreign, non-English characters in Fig. 5 the drawings. The applicant is reminded that the English alphabet must be used for letters except for when another alphabet in customarily used (as is the case for the Greek alphabet for mathematical expressions) (See MPEP § 608.02(V) & 37 CFR § 1.84(p)(2)).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The examiner advises applicant to amend the title to reflect the inventive concept or field of endeavor of the present invention, which is particularly directed to the field of panoramic video playback utilizing a viewer’s pose information.
Claim Objections
Claim 3, 11 & 14 objected to because of the following informalities:
Claims 3 & 14 both recite “the first Euler angle data” and “the second Euler angle data”, however neither of these limitations have been properly introduced as distinct “Euler angle data” earlier in the claim language. The examiner believes these limitations were intended to recite “a first Euler angle data” and “a second Euler angle data”.
Claim 11 recites the following: “correlating the preview posture data and the panoramic image data … to generate the panoramic image data”. The examiner believes the underlined element is intended to recite “panoramic video data”, given that the video data is what’s associated with the posture data to generate the panoramic image data.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-8, & 17-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5 & 17 similarly recite the limitation "the first edited posture data". There is insufficient antecedent basis for this limitation in these claims. Claims 4 and 16 first introduce the limitation “edited posture data”, however it is unclear whether the “first edited posture data” is the same as and/or a subset of the “edited posture data”.
Claims 6-8 & 18-20 are subsequently rejected for their dependence on a claim rejected under 35 U.S.C. § 112(b)
Claims 7 & 19 both recite “the posture data corresponding to each frame…” There is insufficient antecedent basis for this limitation in these claims as “a posture data corresponding to each frame” has not been properly introduced. Given the number of distinct respective forms of “posture data” recited throughout the claim language the examiner suggests clarifying to ensure proper distinction.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 & 9-13 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Desai et al (US 2018/0295284 A1), hereinafter referred to as “Desai”.
Regarding claim 1, Desai teach A video processing method (Desai: the process for tracking the field of view of a content creator 24 as they observe a panoramic image or video [¶0023; Fig. 2]), comprising:
obtaining preview posture data corresponding to a user previewing a panoramic image (Desai: at step 206, the eye tracker 22 captures movement of at least one eye of a content creator 24 (i.e., a user) as they preview video captured at step 204 [¶0024; Fig. 2], proxy streams may be used to preview a video at a reduced resolution [¶0046] – the examiner notes that the movements of the eye (referred to as eye location data) are being interpreted as a form of preview posture data), and send the preview posture data to a recording device (Desai: at step 208, the eye location data is continuously tracked and transmitted to the panoramic camera 4(i.e., a recording device) [¶0024; Fig. 2]);
obtaining panoramic image data from the recording device, the panoramic image data being generated by associating panoramic video data recorded by the recording device with the preview posture data (Desai: at step 210, metadata (i.e., panoramic image data) is generated by embedding the eye location data (i.e., the preview posture data) for each frame of the video content (i.e., panoramic video data) captured from the panoramic camera 4 (i.e., the recording device) [¶0024; Fig. 2]);
extracting the preview posture data and the panoramic video data from the panoramic image data (Desai: at step 212, the eye location data (i.e., preview posture data) and associated video content (i.e., panoramic video data) are accessed (i.e., extracted) from the embedded metadata to automatically pan and/or zoom the video the replicate the field of view of the content creator 24 [¶0024; Fig. 2]); and
playing a video corresponding to the panoramic video data according to the preview posture data (Desai: at step 214, the process repeats to continuously play the video according to the field of view corresponding to the eye location data (i.e., preview posture data) of the content creator 24 [¶0025; Fig. 2]).
As for claim 9, Desai teach A video processing method for a recording device (Desai: the method outlined in Fig. 2 for the panoramic camera 4), comprising:
receiving preview posture data corresponding to a user previewing a panoramic image uploaded by a controller (Desai: at step 208, the eye location data (i.e., preview posture data), is transmitted to the panoramic camera 4 by the eye tracker 22 via the access device 6 (i.e., a controller) [¶0023-24; Fig. 2]);
obtaining recorded panoramic video data (Desai: at step 204, the panoramic camera 4 obtains a stream or recording of video content (i.e., panoramic video data) to display to the content creator 24 [¶0024; Fig. 2]), and
associating the recorded panoramic video data with the preview posture data to generate panoramic image data (Desai: at step 210, eye location data (i.e., preview posture data) is embedded for every frame of the panoramic video content to generate the metadata (i.e., the panoramic image data) [¶0024; Fig. 2]).
Considering claim 10, Desai teach The video processing method according to claim 9 (as described above), further comprising:
sending the panoramic image data to the controller so that the controller plays a video corresponding to the panoramic image data according to the preview posture data (Desai: at step 212, the metadata (i.e., panoramic image data) is transmitted to the access device 6 (i.e., the controller) and automatically updated to adjust to the corresponding location of the content creator 24’s eyes from the eye location data (i.e., preview posture data) as they watch the panoramic video, with the field of view being continuously adjusted in response to this eye location data at step 214 [¶0024-25; Fig. 2]).
Turning to claim 11, Desai teach The video processing method according to claim 9 (as described previously), wherein the obtaining the recorded panoramic video data and associating the panoramic video data with the preview posture data to generate the panoramic image data comprises:
obtaining time information corresponding to the preview posture data and time information corresponding to the panoramic video data (Desai: the eye tracker 22 and panoramic camera 4 may be time-synchronized [¶0023], and the eye location data (i.e., the preview posture data) may be stored in each frame of the video content (i.e., panoramic video data) as metadata (i.e., the panoramic image data)[¶0024] – the examiner notes that frames in a video inherently convey a time signature for each respective frame); and
correlating the preview posture data and the panoramic image data, both of which having the same time information, to generate the panoramic image data (the eye location data (i.e., preview posture data) may be stored in each frame of the video content (i.e., panoramic video data) as metadata (i.e., panoramic image data) [¶0024]).
Regarding claim 12, Desai teach A system (Desai: panoramic camera system 2 [¶0017; Fig. A]), comprising:
a recording device (Desai: panoramic camera 4 [¶0017; Fig. A]); and
a controller (Desai: access device 6 [¶0017; Fig. A]), the controller comprising at least one memory storing executable program codes (Desai: the access device 6 (i.e., the controller) can store data or run computations associated with modules 4A-4C [0017; Fig. A], with data storage available on panoramic camera 4 or the eye tracker apparatus 22 linked to a data storage media 40 [¶0019; Figs. A & 1]) and at least one processor coupled to the at least one memory (Desai: the access device 6 can store data or run computations associated with modules 4A-4C [¶0017; Fig. A], with processors available on camera 4 or the eye tracker apparatus 22 [¶0019; Figs. A & 1]);
wherein the at least one processor, when executing the executable program codes, is configured to:
obtain preview posture data corresponding to a user previewing a panoramic image (Desai: at step 206, the eye tracker 22 captures movement of at least one eye of a content creator 24 (i.e., a user) as they preview video captured at step 204 [¶0024; Fig. 2], proxy streams may be used to preview a video at a reduced resolution [¶0046] – the examiner notes that the movements of the eye (referred to as eye location data) are being interpreted as a form of preview posture data), and send the preview posture data to the recording device (Desai: at step 208, the eye location data is continuously tracked and transmitted to the panoramic camera 4(i.e., a recording device) [¶0024; Fig. 2]);
obtain panoramic image data from the recording device, the panoramic image data being generated by associating panoramic video data recorded by the recording device with the preview posture data (Desai: at step 210, metadata (i.e., panoramic image data) is generated by embedding the eye location data (i.e., the preview posture data) for each frame of the video content (i.e., panoramic video data) captured from the panoramic camera 4 (i.e., the recording device) [¶0024; Fig. 2]);
extract the preview posture data and the panoramic video data from the panoramic image data (Desai: at step 212, the eye location data (i.e., preview posture data) and associated video content (i.e., panoramic video data) are accessed (i.e., extracted) from the embedded metadata to automatically pan and/or zoom the video the replicate the field of view of the content creator 24 [¶0024; Fig. 2]); and
play a video corresponding to the panoramic video data according to the preview posture data (Desai: at step 214, the process repeats to continuously play the video according to the field of view corresponding to the eye location data (i.e., preview posture data) of the content creator 24 [¶0025; Fig. 2]).
When considering claim 13, Desai teach The system of claim 12 (as described above), wherein the recording device is an unmanned aerial vehicle including a panoramic camera (Desai: the apparatus (the panoramic camera system 2) can be mounted on air vehicles, such as a drone [¶0057]), and the controller is a VR pair of glasses, a mobile phone, or a computer (the examiner notes that, due to the disjunctive use of “or” only one limitation in the claimed list requires mapping – Desai: the access device 6 (i.e., the controller) could be in the form of a mobile device such a smart phone [¶0017; Fig. 1]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-3 & 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Desai et al (US 2018/0295284 A1), hereinafter referred to as “Desai”, in view of Li; Adam (US 2017/0185171 A1), hereinafter referred to as “Li”.
Regarding claim 2, Desai teach The video processing method according to claim 1 (described previously), wherein the obtaining the preview posture data corresponding to the user previewing the panoramic image and sending the preview posture data to the recording device (Desai: steps 206 and 208 as described previously [0024; Fig. 2]).
Desai, however, fails to disclose an explicit difference in posture data exceeding a threshold to determine whether or not to update the view of a panoramic image. Li, on the other hand, is analogous art pertinent to the field of endeavor and disclose an orientation estimation system for a virtual reality headset that accounts for only small changes in head posture. More particularly, Li teach comprises:
determining a difference between first preview posture data corresponding to a current video frame and second preview posture data corresponding to a previous video frame (Li: as a user 101 moves to the right, a new view vector 401b (which conveys a first preview posture) is compared to the previous view vector 401a (which conveys a second preview posture data) via a difference in poses measured by the pose change comparator 250 [¶0064-65; Fig. 4]);
in a case that the difference is outside a preset difference range, sending the first preview posture data to the recording device (Li: when the difference in poses is greater than a set threshold, then a new rendering operation based on the updated posture [¶0064-69; Figs. 5 & 6]); and
in a case that the difference is within the preset difference range, stopping sending the first preview posture data to the recording device (Li: when the difference in poses is less than a given threshold, then the pose data is not sent to the scene renderer, and instead the scene is simply re-rendered [¶0064-69; Figs. 5 & 6]).
Li explains that their invention performs comparison of a difference in poses to a threshold to more efficiently decide when a scene should be fully rendered (a time-consuming process) vs re-rendered (a quick and approximate process) [¶0063]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of using difference in pose threshold and apply it to the panoramic viewing platform of Desai to only update a panoramic view when a pose has sufficiently changed, minimizing excess computation.
Turning to claim 3, Desai in view of Li teach The video processing method according to claim 2 (described above), wherein the preview posture data comprises Euler angle data (Li: Euler angles may be used as representations of rotation angles [¶0025] - the examiner notes than any data conveying a rotation around an axis (either roll, pitch, or yaw) can be construed as Euler data), and the determining the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame comprises:
converting the first Euler angle data of the current video frame into a first space vector, and converting the second Euler angle data of the previous video frame into a second space vector (Li: the pose of a user is indicated by a view vector 401, more specifically a new view vector 401b (current) and an a original view vector 401a (previous) [¶0064-65; Fig. 4], a pose change can be a change in an angle of rotation of a user [0068] which can be expressed as Euler angles [¶0025] – the examiner notes that generating a view vector necessitates conversion of angle data into a vectorized form); and
determining a difference between the first space vector and the second space vector (Li: a difference between current (first) and previous (second) view vectors is compared via pose change comparator 250 [¶0065; Fig. 4]).
Li explains that their invention performs comparison of a difference in poses to a threshold to more efficiently decide when a scene should be fully rendered (a time-consuming process) vs re-rendered (a quick and approximate process) [¶0063]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of using difference in pose threshold and apply it to the panoramic viewing platform of Desai to only update a panoramic view when a pose has sufficiently changed, minimizing excess computation.
Concerning claim 14, Desai in view of Li teach The system according to claim 12 (previously described), wherein the obtaining the preview posture data corresponding to the user previewing the panoramic image and sending the preview posture data to the recording device (Desai: steps 206 and 208 as described previously [0024; Fig. 2]).
Desai, however, fails to disclose an explicit difference in posture data exceeding a threshold to determine whether or not to update the view of a panoramic image. Li, on the other hand teach comprises:
determining a difference between first preview posture data corresponding to a current video frame and second preview posture data corresponding to a previous video frame (Li: as a user 101 moves to the right, a new view vector 401b (which conveys a first preview posture) is compared to the previous view vector 401a (which conveys a second preview posture data) via a difference in poses measured by the pose change comparator 250 [¶0064-65; Fig. 4]);
in a case that the difference is outside a preset difference range, sending the first preview posture data to the recording device (Li: when the difference in poses is greater than a set threshold, then a new rendering operation based on the updated posture [¶0064-69; Figs. 5 & 6]); and
in a case that the difference is within the preset difference range, stopping sending the first preview posture data to the recording device (Li: when the difference in poses is less than a given threshold, then the pose data is not sent to the scene renderer, and instead the scene is simply re-rendered [¶0064-69; Figs. 5 & 6]).
Li explains that their invention performs comparison of a difference in poses to a threshold to more efficiently decide when a scene should be fully rendered (a time-consuming process) vs re-rendered (a quick and approximate process) [¶0063]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of using difference in pose threshold and apply it to the panoramic viewing platform of Desai to only update a panoramic view when a pose has sufficiently changed, minimizing excess computation.
As for claim 15, Desai in view of Li teach The system according to claim 14 (as described above), wherein the preview posture data comprises Euler angle data (Li: Euler angles may be used as representations of rotation angles [¶0025] - the examiner notes than any data conveying a rotation around an axis (either roll, pitch, or yaw) can be construed as Euler data), and the determining the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame comprises:
converting the first Euler angle data of the current video frame into a first space vector, and converting the second Euler angle data of the previous video frame into a second space vector (Li: the pose of a user is indicated by a view vector 401, more specifically a new view vector 401b (current) and an a original view vector 401a (previous) [¶0064-65; Fig. 4], a pose change can be a change in an angle of rotation of a user [0068] which can be expressed as Euler angles [¶0025] – the examiner notes that generating a view vector necessitates conversion of angle data into a vectorized form); and
determining a difference between the first space vector and the second space vector (Li: a difference between view vectors is compared via pose change comparator 250 [¶0065; Fig. 4]).
Li explains that their invention performs comparison of a difference in poses to a threshold to more efficiently decide when a scene should be fully rendered (a time-consuming process) vs re-rendered (a quick and approximate process) [¶0063]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of using difference in pose threshold and apply it to the panoramic viewing platform of Desai to only update a panoramic view when a pose has sufficiently changed, minimizing excess computation.
Claim(s) 4 & 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Desai et al (US 2018/0295284 A1), hereinafter referred to as “Desai” in view of Shinohara; Takayuki (US 2019/0219824 A1), hereinafter referred to as “Shinohara.
With respect to claim 4, Desai teach The video processing method according to claim 1, (described previously).
Desai, however, is silent on describing a process for editing posture data when viewing a panoramic image or video. Shinohara, in contrast, is analogous art pertinent to the field of endeavor and outline a process wherein a user can manually adjust the visual line for viewing a panoramic image. Shinohara teach wherein the playing the video corresponding to the panoramic video data according to the preview posture data comprises:
obtaining edited posture data generated after the preview posture data being edited (Shinohara: a user can utilize the left analog stick 77b to change the visual line (i.e., edit the preview posture) [¶0069-71; Fig. 10]); and
playing the video corresponding to the panoramic video data according to the edited posture data and the preview posture data (Shinohara: the panoramic video played is accordance with the adjusted view based on the toggled view from the left analog stick 77b [¶0075-78; Fig. 11]).
Shinohara further describe that the input device 6 is implemented to accommodate a user that may want to manually change the visual line while previewing a panoramic video without having to change their own posture, which may be difficult if a user is seated or if they are wearing a head-mounted display that covers their eyes and prevents them from accurately viewing their surroundings [¶0065-66]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of utilizing the input device of Shinohara to allow a user to manually edit the preview posture data and circumvent changes in view that may be difficult to prompt via eye-movement alone.
As for claim 16, Desai teach The system according to claim 12, (as described previously).
Desai, again, is silent on describing a process for editing posture data when viewing a panoramic image or video. Shinohara, however, teach wherein the playing the video corresponding to the panoramic video data according to the preview posture data comprises:
obtaining edited posture data generated after the preview posture data being edited (Shinohara: a user can utilize the left analog stick 77b to change the visual line (i.e., edit the preview posture) [¶0069-71; Fig. 10]); and
playing the video corresponding to the panoramic video data according to the edited posture data and the preview posture data (Shinohara: the panoramic video played is accordance with the adjusted view based on the toggled view from the left analog stick 77b [¶0075-78; Fig. 11]).
Shinohara further describe that the input device 6 is implemented to accommodate a user that may want to manually change the visual line while previewing a panoramic video without having to change their own posture, which may be difficult if a user is seated or if they are wearing a head-mounted display that covers their eyes and prevents them from accurately viewing their surroundings [¶0065-66]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of utilizing the input device of Shinohara to allow a user to manually edit the preview posture data and circumvent changes in view that may be difficult to prompt via eye-movement alone.
Claim(s) 5-7 & 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Desai et al (US 2018/0295284 A1), hereinafter referred to as “Desai” in view of Shinohara; Takayuki (US 2019/0219824 A1), hereinafter referred to as “Shinohara”, further in view of Tang et al (“Joint Stabilization and Direction of 360 Videos”, ACM Transactions on Graphics, 2018), hereinafter referred to as “Tang”.
Regarding claim 5, Desai in view of Shinohara teach The video processing method according to claim 4, (described previously).
Desai, however, is silent on describing a process for editing posture data when viewing a panoramic image or video. Shinohara, in contrast, is analogous art pertinent to the field of endeavor and outline a process wherein a user can manually adjust the visual line for viewing a panoramic image. Shinohara teach wherein the obtaining the edited posture data generated after the preview posture data being edited comprises:
and
obtaining the first edited posture data generated after the first target preview posture data being edited (Shinohara: a user can utilize the left analog stick 77b to change the visual line (i.e., the first target preview posture) [¶0069-71; Fig. 10]).
Shinohara further describe that the input device 6 is implemented to accommodate a user that may want to manually change the visual line while previewing a panoramic video without having to change their own posture, which may be difficult if a user is seated or if they are wearing a head-mounted display that covers their eyes and prevents them from accurately viewing their surroundings [¶0065-66]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of utilizing the input device of Shinohara to allow a user to manually edit the preview posture data and circumvent changes in view that may be difficult to prompt via eye-movement alone.
While Shinohara describes obtaining an edited preview posture, they fail to disclose a step of determining a particular key frame and associated posture data. Tang, per contra, is analogous art pertinent to the field of endeavor of the present application and describe a 360-motion estimation technique for smoothing camera motion between two keyframes. Tang teach determining a first key frame corresponding to the panoramic image data and first target preview posture data corresponding to the first key frame; (Tang: an essential matrix conveying translational and rotational data (i.e., preview posture data) is estimated for a keyframe [Sec 3.2: Rotation and Translation Estimation - ¶01; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. Therefore, it would have been obvious to implement Tang's joint optimization based on keyframes to the system of Desai in view of Shinohara to ensure smooth stitching of panoramic videos.
Considering claim 6, Desai in view of Shinohara, further in view of Tang teach The video processing method according to claim 5 (outlined above), wherein the playing the video corresponding to the panoramic video data according to the edited posture data and the preview posture data comprises:
and playing the video corresponding to the panoramic video data outside the preset time range according to the preview posture data (Desai: at step 214, the process repeats to continuously play the video according to the field of view of the content creator 24 [¶0025; Fig. 2] – the examiner notes that the lack of a designated time frame recited for Desai implies the video data is played outside a particular time range).
Neither Desai, nor Shinohara disclose designating key frames and utilizing posture data associated with those key frames. Tang, however, teach determining that there is a second key frame within a preset time range corresponding to the first key frame (Tang: the first and last frames of a given time range are selected as keyframes [Sec 3.1: Feature Tracking and Keyframe Selection - ¶02]);
obtaining second edited posture data corresponding to the second key frame (Tang: an essential matrix conveying translational and rotational data (i.e., posture data) is estimated for a keyframe [Sec 3.2: Rotation and Translation Estimation - ¶01; Fig. 1]); and
playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data (Tang: footage is stabilized via keyframe pair rotational and translational direction estimates for a given time frame [Sec 3.2: Rotation and Translation Estimation - ¶01-02; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. One of ordinary skill in the art before the effective filing date would recognize the advantage of implementing the joint stabilization of Tang to the system of Desai in view of Shinohara. This would enable a user to preview a video according to their own visual line until they designate a particular time frame, at which point, joint optimization and keyframe smoothing is performed for video playback according to keyframe translation and rotation.
As for claim 7, Desai in view Shinohara further in view of Tang teach The video processing method according to claim 6 (as described above), wherein the playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data comprises:
performing interpolation calculation on the posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data to obtain interpolation posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame (Tang: rotational and translational motion respective to for all frames across a start and end keyframe are used to inform bilinear interpolation for joint optimization [Sec 3: Method - ¶01; Sec 3.8: 3D Spherical Mesh Warping - ¶03-04; Figs. 1-2 & 7-8]); and
playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the interpolated posture data (Tang: footage is stabilized via keyframe pair rotational and translational direction estimates (derived via bilinear interpolation) for a given time frame [Sec 3.2: Rotation and Translation Estimation - ¶01-02; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. One of ordinary skill in the art before the effective filing date would recognize the advantage of implementing the bilinear interpolation described by Tang to enable smooth frame stitching in the system taught by Desai in view of Shinohara.
With respect to claim 17, Desai in view of Shinohara teach The system according to claim 16, (described previously).
Desai, however, is silent on describing a process for editing posture data when viewing a panoramic image or video. Shinohara, in contrast, teach wherein the obtaining the edited posture data generated after the preview posture data being edited comprises:
and
obtaining the first edited posture data generated after the first target preview posture data being edited (Shinohara: a user can utilize the left analog stick 77b to change the visual line (i.e., the first target preview posture) [¶0069-71; Fig. 10]).
Shinohara further describe that the input device 6 is implemented to accommodate a user that may want to manually change the visual line while previewing a panoramic video without having to change their own posture, which may be difficult if a user is seated or if they are wearing a head-mounted display that covers their eyes and prevents them from accurately viewing their surroundings [¶0065-66]. One of ordinary skill in the art before the effective filing date of the present application would recognize the advantage of utilizing the input device of Shinohara to allow a user to manually edit the preview posture data and circumvent changes in view that may be difficult to prompt via eye-movement alone.
While Shinohara describes obtaining an edited preview posture, they fail to disclose a step of determining a particular key frame and associated posture data. Tang, per contra, is analogous art pertinent to the field of endeavor of the present application and describe a 360-motion estimation technique for smoothing camera motion between two keyframes. Tang teach determining a first key frame corresponding to the panoramic image data and first target preview posture data corresponding to the first key frame; (Tang: an essential matrix conveying translational and rotational data (i.e., posture data) is estimated for a keyframe [Sec 3.2: Rotation and Translation Estimation - ¶01; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. Therefore, it would have been obvious to implement Tang's joint optimization based on keyframes to the system of Desai in view of Shinohara to ensure smooth stitching of panoramic videos.
Considering claim 18, Desai in view of Shinohara, further in view of Tang teach The system according to claim 17 (as described above), wherein the playing the video corresponding to the panoramic video data according to the edited posture data and the preview posture data comprises:
and playing the video corresponding to the panoramic video data outside the preset time range according to the preview posture data (Desai: at step 214, the process repeats to continuously play the video according to the field of view of the content creator 24 [¶0025; Fig. 2] – the examiner notes that the lack of a designated time frame recited for Desai implies the video data is played outside a particular time range).
Neither Desai, nor Shinohara disclose designating key frames and utilizing posture data associated with those key frames. Tang, however, teach determining that there is a second key frame within a preset time range corresponding to the first key frame (Tang: the first and last frames of a given time range are selected as keyframes [Sec 3.1: Feature Tracking and Keyframe Selection - ¶02]);
obtaining second edited posture data corresponding to the second key frame (Tang: an essential matrix conveying translational and rotational data (i.e., posture data) is estimated for a keyframe [Sec 3.2: Rotation and Translation Estimation - ¶01; Fig. 1]); and
playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data (Tang: footage is stabilized via keyframe pair rotational and translational direction estimates for a given time frame [Sec 3.2: Rotation and Translation Estimation - ¶01-02; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. One of ordinary skill in the art before the effective filing date would recognize the advantage of implementing the joint stabilization of Tang to the system of Desai in view of Shinohara. This would enable a user to preview a video according to their own visual line until they designate a particular time frame, at which point, joint optimization and keyframe smoothing is performed for video playback according to keyframe translation and rotation.
Turning to claim 19, Desai in view of Shinohara, further in view of Tang teach The system according to claim 18 (outlined above), wherein the playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data comprises:
performing interpolation calculation on the posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame according to the first edited posture data and the second edited posture data to obtain interpolation posture data corresponding to each frame of panoramic image data between the first key frame and the second key frame (Tang: rotational and translational motion respective to for all frames across a start and end keyframe are used to inform bilinear interpolation for joint optimization [Sec 3: Method - ¶01; Sec 3.8: 3D Spherical Mesh Warping - ¶03-04; Figs. 1-2 & 7-8]); and
playing the video corresponding to the panoramic video data between the first key frame and the second key frame according to the interpolated posture data (Tang: footage is stabilized via keyframe pair rotational and translational direction estimates (derived via bilinear interpolation) for a given time frame [Sec 3.2: Rotation and Translation Estimation - ¶01-02; Fig. 1]).
Tang describes that their motion estimation technique allows them to correct for parallax effects during 360 viewing to enable an immersive "virtual cinema" experience [Sec: Abstract]. One of ordinary skill in the art before the effective filing date would recognize the advantage of implementing the bilinear interpolation described by Tang to enable smooth frame stitching in the system taught by Desai in view of Shinohara.
Allowable Subject Matter
Claims 8 & 20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art neither anticipate nor render obvious the invention as presented in claim 8. More specifically, no prior art specifies performing the particular steps of interpolation calculation under the condition that no second key frames are present within a preset time range and only one key frame is specified (emphasis added). The closest cited prior art, Tang, disclose performing the pose interpolation process recited in claim 8 utilizing video frames for a time start and time end points, but this interpolation requires keyframes specified as the start and end point rather than utilizing a lone key frame as a reference point. Similarly, Yeh et al (“Real-Time Video Stitching”, ICIP, 2017), similarly disclose performing linear interpolation between two keyframes, but fails to disclose a condition wherein only one keyframe is present within a given timeframe.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Yeh et al (“Real-Time Video Stitching”, ICIP, 2017), disclose a real-time video stitching system for panoramic video using keyframe interpolation.
Ju et al (“A panoramic video system by direct manipulation video navigation”, IEEE, 2014) describe a panoramic video viewing system that adapts to a user’s viewing posture.
Li et al (US 2023/0008137 A1) describe an system and method for dynamic field of view selection for 360-degree video based on a detected direction a user is looking.
Yip et al (US 2024/0314287 A1) describe a system for supporting 360 video obtained via drone in accordance with a user’s pose information.
Huang; Cheng (US 2019/0387214 A1) describe a general transmission system for panoramic videos to change the field of view according to a view of a user.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael M. Sofroniou whose telephone number is (571)272-0287. The examiner can normally be reached M-F: 8:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M. Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL M SOFRONIOU/Examiner, Art Unit 2661
/AARON W CARTER/Primary Examiner, Art Unit 2661
Prosecution Insights