DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment / Arguments
103. Applicant’s amendment and response has been considered and, in response, the examiner is not persuaded that the amendments overcome the prior art of record. Applicant’s amendment. The amendment is taught by the prior art: metadata to discern discontinuity is taught by Rogriguez (please note: “metadata” isn’t in Applicant’s specification as filed, and eve the negative limitation Applicant adds is contrary to how applicant’s specification describes abnormality. See Applicant’s published application, paras. 44 and 48), and using timestamps or frame sequence identifiers is also taught by Rodriguez. Accordingly, the 103 rejections are maintained. Please see the remainder of this office action for more details.
Claim Interpretation: 112(f)
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “abnormality detection unit” and “video image selection unit” in claims 1-12.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 8, 9, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Rodriguez (U.S. Patent App. Pub. No. 2018/0053284 A1) in view of Stahl (U.S. Patent App. Pub. No. 2021/0165229).
Regarding claim 1:
Rodriguez teaches: a display system (Fig. 2A: 80 a VR/MR/AR display system) comprising:
a display apparatus of head-mounted type (Fig. 2A: the head mounted glasses, or para. 198: HMD or headset);
a first video processing apparatus including a first computer graphics (CG) generation unit that generates CG (Fig. 2A: 70, local processing module, in combination with para. 198, the local processing module includes a computer vision processor), and configured to output a display-purpose video image to the display apparatus (para. 172, the image generating processor can generate virtual content to be displayed to the user. In one non-limiting example, the processor may convert an image or video associated with the virtual content to a format that can be projected to the user in 3D. For example, in generating 3D content, the virtual content may need to be formatted such that portions of a particular image are displayed at a particular depth plane while others are displayed at other depth planes. This teaches outputting a “display-purpose video image” to the display apparatus. Also, the examiner is interpreting the “image generating processor” of para. 172 to correspond to one embodiment of the “local processing module”, in view of para. 173, which teaches that it can be part of the system (local) or external); and
a second video processing apparatus including a second CG generation unit that generates CG, and configured to communicate with the first video processing apparatus (Fig. 2A: 72, remote processing module, in combination with para. 207, the remote processing module can have a GPU/CPU configuration and perform image processing for delivery to local processing module. Another example of the communication between local and remote processor is in para. 468, and/or illustrated in Fig. 2A or Fig. 2G),
wherein the first video processing apparatus includes: one or more processors (see mapping above, Fig. 2A and para. 198, one or more processors) (alternatively, see Stahl, paras. 53-54, systems or HMD devices with multiple processors (e.g. “a multiprocessor system including several processors (e.g., two, four, eight, or another suitable number)” per para. 54) is known);
one or more memories (Rodriguez, para. 168) including instructions stored thereon that, when executed by the one or processors (e.g. Rodriguez, para. 219, executable instructions stored in memory is known) (alternatively, see Stahl, para. 17), cause the video processing apparatus to function as:
an abnormality detection unit (at least one aspect of the local processor) that detects an abnormality of a video image input from the second video processing apparatus, or an abnormality of a state of connection with the second video processing apparatus…wherein the abnormality detection is based on discontinuity in the video from metadata and is not based on general network conditions, latency, or playback delay (Rodriguez teaches a controller that can modify image data based on “control data” (e.g. Fig. 10: 1010), for display (see paras. 45, 47, 240. Paragraph 240 teaches the controller reading “appended control data” – which corresponds to metadata). This controller can be part of the “first video processing apparatus” (see para. 46, which describes embodiments whereby the controller can be either local or remote, local corresponding to Applicant’s claimed “first video processing apparatus”). The control data can indicate image warping operations to be performed on image data (see para. 246), which is one non-limiting example of detected abnormality of a video image input from the second video apparatus, detected via control data receipt.
Another example of abnormality detection: para. 263: case where the packet header block does not match the expected header…” If a display does detects that the packet header block does not match the expected header, the display can set an error flag and use a previous version of the control packet.” (quoting para. 263) This is a teaching/suggestion of a missing frame, and metadata of the packet header. Other examples of abnormality detection: para. 370, 457, 462, 463. Another example: para. 82: “The method further includes the headset comparing the remote computing module timecode and the headset timecode to identify a first drift. Moreover, the method includes the remote computing module comparing the headset timecode and the remote computing module timecode to identify a second drift” (correction is made, per para. 83), and
a video image selection unit (at least one aspect of the local processor) that selects either one of a first video image using the CG generated by the first CG generation unit and a second video image using the CG generated by the second CG generation unit as a video image to be used for the display-purpose video image, depending on a result of detection by the abnormality detection unit (see above abnormality examples, in ones where processor receives control data that indicates modifying images for display, the second video image will be used for the display-purpose video image. In the case of drift as the detected abnormality, the first video image will be used, with the drift issue corrected).
Regarding: by performing frame-by-frame analysis of video frame continuity using frame sequence identifiers or timestamps included in the video input from the second vide processing apparatus, the reference of Stahl teaches that it is known for HMDs to use a previous frame (i.e. first video image generated by first generation unit) in cases where the HMD does not receive an encoded frame (i.e. a missing video frame), or if the encoded frame is incomplete (i.e. abnormalities are detected) (see Stahl, para. 81, 104; an incomplete frame also teaches/suggests a missing frame); and/or, use the previous frame if the current frame (i.e. frame from the second generation unit) is determined to be missing or incomplete (another example of detected abnormality) (para. 82, 105). This also teaches a frame-by frame basis (see also paras. 49, 82, 104 or claim 22). Re: using frame sequence identifiers or timestamps, timestamps as image or data identifiers is known (Rodriguez, para. 347 or para. 234).
Moreover, like Rodriguez, Stahl also teaches an embodiment whereby HMD and an external device communicatively coupled to the HMD can perform video rendering and/or processing (see para. 37). Modifying the applied references, in view of same, such to include frame by frame analysis, per Stahl, and frame sequence identifiers or timestamps (both, Rodriguez), is all of taught and suggested by the prior art, and would have been obvious and predictable to one of ordinary skill.
It would have been obvious for one of ordinary skill in the art to have modified the applied reference(-s), in view of same, and with or without Stahl, to have included the above, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A). Both references are relevant to correction of errors in video frame processing/receiving/display, and one of ordinary skill in the art would have been further motivated to assemble a comprehensive system/method to correct for frame errors prior to display.
The prior art included each element recited in claim 1, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention. Additional motivation would be to allow for systems to have checks in place to ensure the rendering of desired content.
Regarding claim 2:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the display system according to claim 1, wherein the first video processing apparatus includes a combining unit (Rodriguez, at least one aspect of the local processor) that combines a predetermined video image and CG (Rodriguez, Fig. 33 and “Scene Augmentation” section, beginning at para. 422. This teaches combining a predetermined video image of a 3D scene with CG of, in this example, a character Fig. 33: 2) (see also Rodriguez, para. 426 or claim 1. AR/VR/MR teaches combining predetermined video and CG) (alternatively, Stahl, Fig. 4 and para. 64),
wherein the first video image is a video image obtained by superimposing the CG generated by the first CG generation unit on the predetermined video image (see above mapping to Rodriguez, having the external or second processor perform scene augmentation), and
wherein the second video image is a video image obtained by superimposing the CG generated by the second CG generation unit on the predetermined video image (see above mapping to Rodriguez, having the external or second processor perform scene augmentation), and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
The prior art included each element recited in claim 2, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention. Additional motivation would be to share processing between two processing units, and take advantage of extended/alternate reality processing and experiences.
Regarding claim 3:
Rodriguez and/or Stahl teach: the display system according to claim 2, wherein the second video processing apparatus includes a combining unit (Rodriguez, at least one aspect of the external processor)) configured to generate a composite image by overlaying CG generated by the second CG generation unit onto the predetermined video image using image blending (see mapping to claim 2 re: augmentation) (note: claim interpretation, “combining” the predetermined video with CG corresponds to “superimposing” of the CG and predetermined video, as in claim 2) (alternatively, Stahl, para. 51, overlaying to generate composite images is known),; and
wherein the video image input from the second video processing apparatus is a composite image generated by the combining unit (Stahl, para. 51, composite image).
Note: claim interpretation [Wingdings font/0xE0] Applicant’s specification as filed does not contain the term “blend” or “blending” in any iteration. The examiner is interpreting the claimed image blending to be redundant, or part of the overlaying to generate a composite image.
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of same, to have obtained the above, motivated to provide extended reality to users and image combinations.
Regarding claim 4:
Stahl teaches: the display system according to claim 2, wherein the display apparatus includes an image pickup unit, and wherein the predetermined video image is a video image picked up by the image pickup unit (Stahl, para. 10 and Fig. 4: “a method of operation for a mixed reality system in which rendered frames of virtual content received from the base station are composited with frames from the HMD scene cameras on the HMD,” see also claims 29, 35, 40 and para. 30, 40-45, for more teaching. The camera is the “image pickup unit”).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of same, to have obtained the above, motivated to provide extended reality to users relevant to their current or surrounding scene.
Regarding claim 8:
Rodriguez and/or Stahl teach: the display system according to claim 1, wherein the video image input from the second video processing apparatus is the CG generated by the second CG generation unit (see mapping to claim 1; claim 8 is mapped in claim 1, and re-iterates that the input from the second apparatus (as reviewed by abnormality detection unit) is the CG generated by second apparatus, as mapped/recited in claim 1 as part of the functions performed by second apparatus).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of same, to have obtained the above, motivated to ensure that the correct or proper images/video are displayed.
Regarding claim 9:
Rodriguez and/or Stahl teach: the display system according to claim 1, wherein the video image selection unit selects the second video image as the display-purpose video image, in a case where no abnormality is detected by the abnormality detection unit (see mapping to claim 1, no changes need to be made, display the input from second apparatus), and
wherein the video image selection unit selects the first video image as the display-purpose video image, in a case where an abnormality is detected by the abnormality detection unit (this is an example of a shift abnormality (Rodriguez), or missing/incomplete frames (Stahl). See mapping to claim 1).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of same, to have obtained the above, motivated to ensure that the correct or proper images/video are displayed.
Regarding claim 11:
Stahl teaches: the display system according to claim 1, wherein the first video processing apparatus and the second video processing apparatus each include a recording/reproducing unit that records a video image and reproduces a video image (Stahl, para. 39-41 and claim 29, HMD (first apparatus) has a recording unit that records and reproduced; Stahl, para. 59, base station (second apparatus) can be a desktop computer, notebook or laptop computer, pad or tablet device, smartphone, hand-held computing device, any of these include a camera (recording/reproducing apparatus)) using a storage unit (Stahl, para. 55, memory of HMD, to store images captured), and para. 59, the devices of second apparatus (base station) also have memory)).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of same, to have obtained the above, motivated to make use of known hardware architecture to perform desired tasks.
Regarding claim 12: see also claim 1.
The features of claim 12 are an alternate embodiment, or similar to those of claim 1, as mapped below; the same rationale for rejection applies:
a video processing apparatus (first video processing apparatus of claim 1) configured to output a display-purpose video image to a display apparatus of head-mounted type (claim 1, display apparatus), the video processing apparatus comprising:
one or more processors; one or more memories including instructions stored thereon that, when executed by the one or more processors, cause the video processing apparatus to function as (claim 1):
a first computer graphics (CG) generation unit configured to generate CG (claim 1);
an abnormality detection unit configured to detect an abnormality of a video image input from another video processing apparatus including a second CG generation unit that generates CG, or an abnormality of a state of connection with the other video processing apparatus by performing frame-by-frame analysis of video frame continuity using frame sequence identifiers or timestamps included in the video input from the second vide processing apparatus, wherein the abnormality detection is based on discontinuity in the video from metadata and is not based on general network conditions, latency, or playback delay (claim 1, the “another video processing apparatus” corresponds to the “second view processing apparatus” of claim 1); and
a video image selection unit configured to select either one of a first video image using the CG generated by the first CG generation unit and a second video image using the CG generated by the second CG generation unit as a video image to be used for the display-purpose video image, depending on a result of detection by the abnormality detection unit (claim 1),
wherein the abnormality detection unit is configured to detect missing video frames in the video image input from the other video processing apparatus on a frame-by-frame basis (claim 1).
Regarding claim 13: see claim 12.
The “control method for a video processing apparatus” of claim 13 corresponds to the functions performed by the video processing apparatus of claim 12. The same rationale for rejection applies.
Regarding claim 14: see also claims 1 and 13.
Rodriguez teaches: a non-transitory computer-readable storage medium storing a program for causing a computer to execute a control method (para. 543) for a video processing apparatus (system of claim 1). The control method for a video processing apparatus corresponds to the control method of claim 13; the same rationale for rejection applies.
Claim(s) 7 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Rodriguez in view of Stahl and further in view of Dimitrov (U.S. Patent App. Pub. No. 2019/0164518).
Regarding claim 7:
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained: the display system according to claim 1, wherein the abnormality detection unit determines that the video image is stopped, in a case where missing in the video image input from the second video processing apparatus continues for a predetermined period, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Dimitrov teaches a “system …generating image frames associated with the application at a frame rate, and a client which displays the image frames on a display that has a predetermined refresh rate and which monitors arrival times of the image frames in relation to the predetermined refresh rate” (Abstract). Dimitrov teaches that it is known for “stutter” to occur “when writing into the display buffer is delayed with respect to reading according to the refresh rate of the display, and the display reads and redraws stale (already previously displayed) content from the buffer.” (para. 10. See also paras. 62-67). The identification of stutter (paras. 62-67) by Dimitrov teaches Applicant’s claim 7. Modifying the applied refs, in view of Dimitrov, such to include stuttering as an abnormality case (where the missing video continues for a time period), is all of taught and suggested by Dimitrov, and obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention. Additional motivation would be to be aware of and monitor/address issues that can hinder desired frame rate.
Regarding claim 10:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the display system according to claim 1, further comprising a buffer configured to absorb a timing difference when the video image selection unit switches between the first video image and the second video image, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Dimitrov teaches a buffer that can hold frames prior to display (see para. 66-67). This teaches the buffer of claim 10, to absorb a timing difference when switching between first and second video image (i.e. one that arrives from the second apparatus). Modifying the applied references, in view of same, to have included the buffer capabilities of Dimitrov, such to alleviate timing discrepancies (which all three references are motivated to achieve), would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613