DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-10, 12-15, 17-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Corcoran (US 20180332219 A1) in view of Raveendran (US 20110032328 A1).
As per claim 1, Corcoran teaches the claimed:
1. A method of simulating perception of a user at an incident, comprising:
receiving first video data from a first camera worn by the user at the incident;
receiving second video data from a second camera worn by the user at the incident; and
(Corcoran [0010]: “A helmet permits a user wearing the helmet to receive a first field of view in the surroundings based on optical information received with the user's natural vision directly from the surroundings without digital video processing. A plurality of camera units are mounted about the helmet to generate multiple channels of video data.”
Corcoran teaches the plurality of cameras, corresponding to a first and second camera, worn on a helmet that faces a user’s surroundings, or a user in an incident. These cameras also generate video data for each of the plurality of cameras.)
Corcoran alone does not explicitly teach the remaining claim limitations.
However, Corcoran in combination with Raveendran teaches the claimed:
combining the first video data and the second video data to provide simulated video data,
(Raveendran [0012]: “The apparatus also includes at least one interface that forwards the 3D video data to the 3D display device, and a human visual system (HVS) feedback module that determines, while the transformation module transforms the video data, one or more metrics using an HVS model that reflects a quality of 3D visualization of the generated 3D video data with respect to a human visual system and, again while the transformation module transforms the video data, reconfigures the one or more modules based on the determined one or more metrics to refine the generation of the 3D video data.”
Raveendran [0118]: “transformation module 36 may include multiple 2D-to-3D processing module (e.g., multiple 2D-to-3D processing modules may be invoked) to process each frame of a different stream that was acquired or captured at the same time from multiple capture systems.”
Raveendran [0119]: “With respect to multi-view streams where each view has been acquired at the same time with different camera units (each camera located at a different point in space with the same or different camera parameters), 2D-to-3D processing unit 58 may include multiple cameras modeling units similar to lens modeling unit 88A. Each of these lens modeling units 88A process a frame of a different multi-view, where each of these frames are captured at the same time.”)
wherein the simulated video data indicates the perception of the user at the incident.
(Raveendran [0100]: “HVS feedback module 40 may configure various modules 56-64 of transformation module 36 to correct a viewing position so as to improve stereo perception for a volumetric display.”
Raveendran [0107]: “HVS feedback module 40 may modulate the range of expansion and compression based on perceptibility or sensitivity of a given viewer's HVS as defined by one of user preferences 78.”
Corcoran [0010]: “Processing circuitry generates a composite field of view from some of the channels of video data. … the composite field of view subtends a wide field of view angle, and the circuitry displays images of the scene based on selectable field of view angles of the scene, with the images derived from one or more of the channels of video data.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the generation of 3D video data as taught by Raveendran with the system of Corcoran in order to enable a more complete capture of scene geometry and motion to result in improved 3D representation and visualization.
As per claim 2, Corcoran teaches the claimed:
2. The method of claim 1, wherein the second video data comprises a higher resolution than the first video data.
(Corcoran [0064]: “the user may next select a second enhancement mode to initiate a higher resolution scan of one or both peripheral camera vision regions.”
Corcoran teaches the cameras are adjustable to enhance resolution in certain cameras, thus selecting a camera to have a higher resolution than another camera will result in the first video data having a higher resolution than the other.)
As per claim 14, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
As per claim 3, Corcoran teaches the claimed:
3. The method of claim 1, wherein the simulated video data comprises the second video data overlaid within a portion of the first video data.
(Corcoran [0048]: “Cameras in the first alternate embodiment may each provide a view somewhat larger than 180° to provide a modest overlap between fields of view that assures continuity in a full 360° composite view of the surroundings.”)
As per claim 15, this claim is similar in scope to limitations recited in claim 3, and thus is rejected under the same rationale.
As per claim 5, Corcoran teaches the claimed:
5. The method of claim 1, wherein the first camera captures a first field of view and the second camera captures a second field of view, and wherein the first field of view is wider than the second field of view.
(Corcoran [0010]: “Each camera channel captures a different field of view of a scene in a region … The cameras may include adjustable settings with the circuitry controlling the camera settings for acquiring the video data. The adjustable settings include optical field of view angles”.
Corcoran teaches each camera captures a different field of view and that each camera can be adjustable to have different angles of the field of view, indicating that the first camera can have a wider field of view than the second based on the camera adjustments.)
As per claim 17, this claim is similar in scope to limitations recited in claim 5, and thus is rejected under the same rationale.
As per claim 6, Corcoran teaches the claimed:
6. The method of claim 1, wherein the perception of the user comprises at least one of a head orientation of the user, a gaze of the user, or a lighting adaptation of an eye of the user.
(Corcoran [0038]: “Two lens imaging systems, each having identical characteristics and settings, including the same field of view angle through which optical information is received, may provide two different fields of view based on differences in position or orientation of the optical axis of each.”
Corcoran teaches the orientation of the optical axis of the cameras, and since the cameras are a helmet on the user, it can be said that the field of view or perception of the user comprises of the head orientation of the user.)
As per claim 18, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
As per claim 7, Corcoran and Raveendran teaches the claimed:
7. The method of claim 1, wherein combining the first video data and the second video data comprises matching visual content of a portion of the first video data with visual content of the second video data.
(Raveendran [0118]: “These descriptors will then be used to set the correspondence between the feature points in different frames, where the location of correspondent points will be used along with the projective geometry camera model implemented by camera modeling unit 86A to extract the depth values.”
Raveendran [0119]: “With respect to multi-view streams where each view has been acquired at the same time with different camera units (each camera located at a different point in space with the same or different camera parameters), 2D-to-3D processing unit 58 may include multiple cameras modeling units similar to lens modeling unit”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the camera modeling unit as taught by Raveendran with the system of Corcoran in order to generate a proper alignment of the scene content to increase consistency and accuracy of the combined video representation.
As per claim 8, Corcoran teaches the claimed:
8. The method of claim 1, wherein combining the first video data and the second video data comprises modifying a shape of a field of view captured in the second video data to provide the simulated video data.
(Corcoran [0010]: “The system may further include a display unit, and the circuitry may include a programmable processing unit in which the circuitry stores the composite field of view in the memory, presents portions of the composite field of view on the display unit, and adjusts scene field of view angles for image presentation on the display unit.”)
As per claim 9, Corcoran teaches the claimed:
9. The method of claim 1, wherein the second video data comprises a shorter exposure time than the first video data.
(Corcoran [0010]: “The cameras may include adjustable settings with the circuitry controlling the camera settings for acquiring the video data. The adjustable settings include optical field of view angles, camera exposure speed”
Corcoran [0046]: “According to an embodiment of the invention, camera settings are individually adjustable, and exposure settings of cameras having fields of view facing brighter light (e.g., sunlight) are higher than the exposure settings of cameras for which the fields of view face lower light levels, e.g., mostly shaded or shadow regions.”
Corcoran teaches each camera’s exposure speed can be individually adjusted and thus the camera that captures the second video data can be adjusted to have a shorter exposure time compared to the first video data.)
As per claim 10, Corcoran teaches the claimed:
10. The method of claim 1, wherein the second camera comprises a slower algorithm for adapting to brightness conditions at the incident than the first camera.
(Corcoran [0010]: “The cameras may include adjustable settings with the circuitry controlling the camera settings for acquiring the video data. The adjustable settings include optical field of view angles, camera exposure speed”.
Corcoran teaches that each camera settings for camera exposure speed is adjustable, thus the second camera can be adjusted in a way that is slower to adapting to brightness at the scene compared to the first camera.)
As per claim 12, Corcoran and Raveendran teaches the claimed:
12. The method of claim 1, further comprising:
receiving third video data from a third camera worn by the user at the incident, the third video data tracking movement of an eye of the user at the incident; and
(Raveendran [0098]: “If eye tracking is available through display device 16, HVS
feedback module 40 may receive this feedback via the same interface by which HVS feedback module 40 receives 3D video content 44. This mechanism may be an external device coupled to display device 16. Alternatively, this eye tracking mechanism may be integrated within display device 16.”)
modifying the simulated video data based at least in part on the third video data.
(Raveendran [0100]: “HVS feedback module 40 may configure various modules 56 – 64
of transformation module 36 to correct a viewing position so as to improve stereo perception for a volumetric display.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the eye tracking system as taught by Raveendran with the system of Corcoran in order to correct a viewing position of the user and to improve the stereo perception of the 3D scene for the user.
As per claim 20, this claim is similar in scope to limitations recited in claim 12, and thus is rejected under the same rationale.
As per claim 13, the reasons and rationale for the rejection of claim 1 is incorporated herein. In particular, only additional features unique to claim 13 that were not present in claim 1 will be explicitly addressed here. (strikethrough taught in claim 1)
As per claim 13, Corcoran teach the claimed:
13. A system for simulating perception of a user at an incident, comprising:
a first camera, the first camera worn by the user at the incident and configured to capture first video data;
a second camera, the second camera worn by the user at the incident and configured to capture second video data;
(Corcoran [0010]: “A helmet permits a user wearing the helmet to receive a first field of view in the surroundings based on optical information received with the user's natural vision directly from the surroundings without digital video processing. A plurality of camera units are mounted about the helmet to generate multiple channels of video data.”)
a non-transitory computer-readable medium storing computer-readable instructions for providing simulated video data, the simulated video data indicating perception of the user at the incident; and
a processor communicatively coupled to the first camera, the second camera, and the computer-readable medium, the processor further configured to execute the instructions, wherein the instructions, when executed, cause the processor to perform operations comprising:
(Corcoran [0054]: “The central processing and control unit 14 includes a multi-channel two-directional data transmission interface 36 which receives parallel streams of video data from all of the imaging modules 24 via high speed data lines 50. Frames of the video data are received into a central processing unit (CPU) 38 comprising a processor 40, microprocessor memory 42, video memory 44 L, video memory 44 R and data storage 48 containing data and executable software which runs on the processor 40.”
Corcoran [0051]: “each camera 16 is part of a processor-based imaging module”)
Claims 4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Corcoran in view of Raveendran and in further view of Kang (CN 110213626 A).
As per claim 4, Corcoran and Raveendran alone does not explicitly teach the claimed limitations.
However, Corcoran and Raveendran in combination with Kang teaches the claimed:
4. The method of claim 1, wherein combining the first video data and the second video data comprises smoothing a resolution of the simulated video data between a first portion of the simulated video data corresponding to the first video data and a second portion of the simulated video data corresponding to the second video data.
(Kang (page 8, line 19-21, 26-29): “by receiving the first video data and the second video data of the target video determined according to the network bandwidth, it is possible to ensure the smoothness of video playback … By recombining high-resolution video and low-resolution video, the combined video can achieve the viewing effect of higher-resolution video, which can be used in network bandwidth. Under bad conditions, it not only ensures the smoothness of video playback, but also improves the visual video viewing effect.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the smoothing of resolution as taught by Kang with the system of Corcoran as modified by Raveendran in order to improve and achieve the visual video viewing effect of the higher resolution video and ensuring the smoothness of the video.
As per claim 16, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Corcoran in view of Raveendran and in further view of Bleyer (US 20210392312 A1).
As per claim 11, Corcoran and Raveendran alone do not explicitly teach the claimed limitations.
However, Corcoran and Raveendran in combination with Bleyer teaches the claimed:
11. The method of claim 1, wherein the first video data is video stabilized and the second video data is not video stabilized.
(Bleyer [0006]: “Embodiments disclosed herein relate to systems, devices (e.g., wearable devices, hardware storage devices, etc.), and methods for aligning and stabilizing images generated by an integrated camera that is physically mounted to a computer system (e.g., perhaps an HMD) with images generated by a detached camera that is physically unmounted from the computer system.”
Bleyer teaches the integrated and detached camera, where the integrated camera data has stabilization and is aligned and stabilized with the detached camera data that is not stabilized. Therefore, Bleyer teaches the first camera containing video data that is stabilized, and the second camera containing video data that is not stabilized.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the alignment of the stabilized and not stabilized video data as taught by Bleyer with the system of Corcoran as modified by Raveendran in order to enable corresponding scenes to be matched more accurately and improve the consistency in the frames and motions.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Corcoran in view of Raveendran and in further view of Arnold (US 20180249087 A1).
As per claim 19, Corcoran and Raveendran alone do not explicitly teach the claimed limitations.
However, Corcoran and Raveendran in combination with Arnold teaches the claimed:
19. The system of claim 13, wherein the first camera comprises a torso-mounted camera and the second camera comprises a head-mounted camera.
(Arnold [0006]: “In a first embodiment a system is incorporated into a vest or garment that is designed to be worn over the torso.”
Arnold [0008]: “In a further embodiment, the camera elements of the system are mounted on a protective helmet.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the torso and head mounted cameras as taught by Arnold with the system of Corcoran as modified by Raveendran in order to capture complementary view of a scene that allows the output to comprise a more complete representation of the user’s environment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA SUO whose telephone number is (571) 272-8387. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA SUO/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights