Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 2, 4, 5, 7, 8, 9, 14, 17, 18, 33 and 34 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Powell US 2023/0230210.
Re claim 1 Powell discloses
A videoconferencing system for adjusting perspective views using adaptive image warping, the system comprising: an image warping unit including at least one processor programmed to (see paragraph 3):
receive an overview video stream from a camera in the videoconferencing system (see paragraph 1 and 21 note that a camera provides video conferencing image data);
determine, based on analysis of at least one test frame from the overview video stream, at least one region of interest represented within the at least one test frame (see paragraph 52 53 and figure 15 note that a region of interest may be cropped);
determine one or more indicators of actual camera perspective relative to the at least one region of interest (see paragraph 34 and 54 note that the tile angle of the camera is determined according to a tilt sensor, the examiner note that this tilt is relative to the image as a whole including the region of interest);
determine a target camera perspective relative to the at least one region of interest, wherein the target camera perspective is different from the actual camera perspective (see paragraph 23 note the target perspective is compensated for the tilt so that vertical objects appear straight see also figure 13 note that in element 1310 the tilt of 17 degrees is corrected for);
determine at least one image transformation based on a difference between the actual camera perspective and the target camera perspective (see paragraph 45-48 and 54-55 note that a projection is determined to warp the image using the tilt angle see also figure 14 step 1408);
apply the at least one image transformation to one or more subframe regions of a plurality of image frames of the overview stream to generate at least one image warped primary video stream (see paragraph 45-48 and 54-55 note that a projection is determined to warp the image to the desired perspective this projection is applied on a pixel by pixel basis see also figure 14 step 1408 note that corrected image is output);
and cause the at least one image warped primary video stream to be shown on a display (see paragraph 62 and 68 note that the distortion corrected images may be displayed).
Re claim 2 Powell discloses wherein the at least one region of interest includes a videoconference participant (see figure 13 note that multiple video conference participants are in the region of interest).
Re claim 4 Powell disclose wherein the at least one region of interest includes two or more videoconference participants. (see figure 13 note that multiple video conference participants are in the region of interest).
Re claim 5 Powell discloses wherein the at least one region of interest includes one or more objects (see figure 13 note that the region of interest includes multiple objects).
Re claim 7 Powell disclose wherein the at least one region of interest is delineated by a rectangular boundary within the at least one test frame (see paragraph 13 note that the region of interest is delineated by a rectangular boundary).
Re claim 8 Powell discloses wherein the at least one region of interest is delineated by a polygonal boundary within the at least one test frame (see paragraph 13 note that the region of interest is delineated by a rectangular boundary).
Re claim 9 Powell discloses wherein the polygonal boundary is a quadrangle (see paragraph 13 note that the region of interest is delineated by a rectangular boundary).
Re claim 14 Powell discloses wherein the one or more indicators of actual camera perspective relative to the at least one region of interest are determined, at least in part, based on an output of a sensor separate from the camera (see paragraph 31 “Further, in some examples, tilt angle data from a tilt sensor 213 may be used to select a distortion correction projection 212 to apply” note that a tilt sensor may be used to determine the tilt of the camera).
Re claim 17 Powell discloses wherein the one or more indicators of actual camera perspective relative to the at least one region of interest are determined, at least in part, based on a predetermined camera model. (See paragraph 31 “Further, tilt parameter may be provided by use of dedicated mechanical camera mount having a known tilt angle with reference to the lens optical axis.” Note the camera model i.e. with a known tilt value mat indicate the tilt value).
Re claim 18 Powell discloses wherein the predetermined camera model indicates at least one of a field of view angle, a pitch value, a tilt value, a roll value, a yaw value, or a pan value associated with the camera (see paragraph 31 Further, tilt parameter may be provided by use of dedicated mechanical camera mount having a known tilt angle with reference to the lens optical axis.” Note the camera model i.e. with a known tilt value mat indicate the tilt value).
Re claim 33 Powell discloses wherein the image warping unit is located on the camera (see paragraph 58 note that the system may involve a computing device incorporating the camera).
Re claim 34 Powel discloses wherein the image warping unit is remotely located relative to the camera (see paragraph 57 58 and 67 note that the service i.e. application may be located on a server).
Claim(s) 1,3, 11-13, 21-25 and 30 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhou US 20220335579 A1.
Re claim 1 Zhou discloses
A videoconferencing system for adjusting perspective views using adaptive image warping, the system comprising:
an image warping unit including at least one processor programmed to (see paragraph 101-102 note that device may be implemented by a processor):
receive an overview video stream from a camera in the videoconferencing system (see paragraph 103 and 8 note that images are received and processed);
determine, based on analysis of at least one test frame from the overview video stream, at least one region of interest represented within the at least one test frame (see paragraph 39 and figure 1 note that regions 125A and 125 B are found);
determine one or more indicators of actual camera perspective relative to the at least one region of interest; (see paragraph 80 for example “ROI 435 is centered at a location that may be represented by a central point 440, having spherical coordinates of (θ.sub.yaw, θ.sub.pitch), that is, a yaw rotation of θ.sub.yaw degrees and a pitch rotation of θ.sub.pitch degrees from the central point 430” note that the initial coordinates of the face correspond to the initial perspective )
determine a target camera perspective relative to the at least one region of interest wherein the target camera perspective is different from the actual camera perspective; (see paragraph 55 virtual camera target see also paragraph 80 note that the change in perspective to for example the camera center involves reprojection of the face),
determine at least one image transformation based on a difference between the actual camera perspective and the target camera perspective (see paragraph 80 and 81 note that a reprojection is determined and the face is reprojected into a virtual camera view);
apply the at least one image transformation to one or more subframe regions of a plurality of image frames of the overview stream to generate at least one image warped primary video stream (see paragraph 80 and 81 note that a reprojection is determined face region is reprojected into a virtual camera view)
and cause the at least one image warped primary video stream to be shown on a display (see paragraph 63 and claim 15 note that an output image may be displayed).
Re claim 3 Zhou discloses wherein the at least one region of interest includes: a first region of interest including a first videoconference participant and at least a second region of interest including a second videoconference participant (see paragraph 34 note that multiple faces in the image may be determined and see paragraph 8 note that faces may be used in a video conferencing application).
Re claim 11 Zhou discloses wherein the one or more indicators of actual camera perspective relative to the at least one region of interest include coordinates, in a reference frame of the camera, of at least one point associated with a determined boundary delineating a representation of the at least one region of interest in the at least one test frame (see paragraph 49 note that box coordinates of the ROI including a participant may by used to transform from the input image space coordinates [i.e. actual camera perspective] the framing space coordinates see also paragraph 80).
Re claim 12 Zhou discloses wherein the one or more indicators of actual camera perspective relative to the at least one region of interest include coordinates, in a reference frame of the camera, of at least one point associated with an object or videoconference participant located in the at least one region of interest (see paragraph 49 note that box coordinates of the ROI including a participant may by used to transform from the input image space coordinates [i.e. actual camera perspective] to the output image space see also paragraph 80).
Re claim 13 Zhou discloses wherein the one or more indicators of actual camera perspective relative to the at least one region of interest include coordinates, in a reference frame of the camera, for each of a plurality of pixels included in a representation of the at least one region of interest (see paragraph 80 and 81 and figure 4 . Once the desired portion of the input image FOV (460) is determined, it may be rotated back to the center of the camera's FOV and re-projected (arrow 485) into a planar output image coordinate system (also referred to herein as “output image space”) to generate a planar and perspective-corrected output image 490, including the newly-centered (and distortion-corrected) ROI 495, note that the coordinates entire ROI are reprojected and rotated from the input perspective [ actual camera perspective] to the output perspective).
Re claim 21 Zhou discloses wherein the target camera perspective includes a plurality of different target camera perspectives, each associated with one or more corresponding image transformations (see paragraph 80 note that reprojection may be performed for a face and paragraph 98 and 99 and figure 6 note that processing may be performed for each of two regions of interest the examiner notes paragraph 80 applied to each of the different faces would necessarily result in two different target corrections).
Re claim 22 Zhou discloses wherein the target camera perspective includes a plurality of different target camera perspectives, each associated with one or more corresponding image transformations (see paragraph 80 note that reprojection may be performed for a face and paragraph 98 and 99 and figure 6 note that processing may be performed for each of two regions of interest the examiner notes paragraph 80 applied to each of the different faces resulting in two perspective corrected images).
Re claim 23 Zhou discloses wherein each of the one or more corresponding image transformations is applied to a same number of frames from among the plurality of image frames of the overview video stream (see paragraph 8 and 63 and paragraph 98 and 99 note distortion correction is performed for each frame for multiple regions of interest this would result in performing the transformation the same number of times).
Re claim 24 Zhou discloses wherein each of the one or more corresponding image transformations is applied to a linearly varying number of frames from among the plurality of image frames of the overview video stream (see paragraph 56 and 57 “Instead, it may take the virtual camera a number of frames to reach its target location, moving only a portion of the way to its target location with each successive frame.“ note that perspective may be intelligently shifter over a number of frames see paragraph 60 “Likewise, constant constraints on acceleration and/or velocity may be used to ensure that the panning is never too jerky or too fast.” Note that constant velocity would be a linear change over the number of frames).
Re claim 25 Zhou discloses wherein each of the one or more corresponding image transformations is applied to a non-linearly varying number of frames from among the plurality of image frames of the overview video stream (see paragraph 56 and 57 “Instead, it may take the virtual camera a number of frames to reach its target location, moving only a portion of the way to its target location with each successive frame.“ note that perspective may be intelligently shifter over a number of frames see paragraph 60 “Likewise, constant constraints on acceleration and/or velocity may be used to ensure that the panning is never too jerky or too fast.” Note that acceleration would involve a change of velocity and the distortion correction would occur non-linearly over the number of frames).
Re claim 30 Zhou discloses, wherein the target camera perspective is along a line substantially normal to a center point associated with an object or videoconference participant represented in the at least one test frame. (See paragraph 80 “Central point 430 may also be thought of as the optical center of the input camera. In the example of image 410, there is a region of interest (e.g., face 435) that the system wishes to “point” the virtual camera at, e.g., for the purpose of creating a planar, perspective-corrected output image from the wide input image's FOV for use in a videoconferencing application or the like” note that the target camera perspective is center point of the camera. )
Claim(s) 1, 6, 27 and 31 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Anderson US 2022/0334638.
Re claim 1 Anderson discloses
A videoconferencing system for adjusting perspective views using adaptive image warping, the system comprising:
an image warping unit including at least one processor programmed to (see paragraph 11 note that the gaze correction may be implemented by a processor): receive an overview video stream from a camera in the videoconferencing system;(see paragraph 29 note that the method may be applied to a video stream of a video unit)
determine, based on analysis of at least one test frame from the overview video stream, at least one region of interest represented within the at least one test frame (see paragraph 37 note that eyes are detected);
determine one or more indicators of actual camera perspective relative to the at least one region of interest (see paragraph 37 note that the direction of the eyes is detected relative to the camera are determined);
determine a target camera perspective relative to the at least one region of interest, wherein the target camera perspective is different from the actual camera perspective (see paragraph 38 note that the target perspective of the camera to the eyes is such that the eyes appear to be looking at the camera);
determine at least one image transformation based on a difference between the actual camera perspective and the target camera perspective; apply the at least one image (see paragraph 38-40 or 56 note that a correction is determined to make it appear as if the user is looking at the camera);
transformation to one or more subframe regions of a plurality of image frames of the overview stream to generate at least one image warped primary video stream (see paragraph 56 note that the eyes may be warped to appear as if they look at the camera); and cause the at least one image warped primary video stream to be shown on a display (see paragraph 58 “gaze corrected frames 214 are presented by the participant device(s) of the selected participants, the gaze corrected frames 214 can simulate eye contact between the user 104 and the selected participant(s)” note that the corrected frames can be presented i.e. displayed to the users).
Re claim 6 Anderson discloses wherein the at least one region of interest represented with the at least one test frame is determined by a trained neural network (see paragraph 54 note that the eyes are determined based on the neural network).
Re claim 27 Anderson discloses provide at least one trained neural network configured to receive the at least one test frame as an input, determine the at least one region of interest represented in the test frame (see paragraph 54 note that an neural network is trained to analyze the images and locate the face and eyes ), and output the one or more indicators of actual camera perspective (see paragraph 54 and 57 and 37 note that orientation of the face and the eyes relative to the camera is determined using a neural network )
Re claim 31 Anderson discloses wherein the target camera perspective is determined based on input received from a user of the videoconferencing system (See paragraph 44 and 41 note that the gaze management circuitry determines whether or not to perform the correction to correct the eyes looking at the camera this effectively set the target perspective. Since the Gazing is done by the user it is user input).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Powell US 2023/0230210 in view of Schaefer US20220292801.
Re claim 6 Powell discloses determining a region of interest (see paragraph 52 53 and figure 15 note that a region of interest may be cropped). Powell does not expressly disclose wherein the at least one region of interest represented with the at least one test frame is determined by a trained neural network. Schaefer disclose wherein the at least one region of interest represented with the at least one test frame is determined by a trained neural network. (see paragraph 24-30 note that neural networks are used to determine regions of interest and a frame surrounding them.) The examiner notes that Powell merely describes cropping a region of interest without much explanation on how that region is found. The frame determination of Schaefer could have easily been substituted for the cropping of Powell. One of ordinary skill in the art could have easily used the method of Schaefer to determine the region of interest in Powell and result i.e. Powell is applied to the region of interest, would have been predictable. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Powell and Schaefer.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou US 20220335579 A1 in view of Han US 2022/0353438.
Re claim 10 Zhou does not expressly disclose wherein the at least one region of interest is delineated by a boundary in the at least one test frame that traces an outline of a perimeter associated with a representation of at least one videoconference participant. Han discloses wherein the at least one region of interest is delineated by a boundary in the at least one test frame that traces an outline of a perimeter associated with a representation of at least one videoconference participant (see paragraph 57 note that an outline of a user is determined based on a facial recognition algorithm also see figure 6). The motivation to combine is to allow the removal of pixels outside of the area defined by the user (see paragraph 43). One of ordinary skill in the art could have easily used the teachings of Han to modify Zhou to determine a region of interest of the user by outlining a user. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhou and Han to reach the aforementioned advantage.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Powell US 2023/0230210 in view of Dao US 2024/0214520.
Re Claim 15 Powell discloses the camera or device incorporating the camera also may include a tilt sensor (e.g. an inertial motion sensor, rotational encoder incorporated into the pivot joint, etc.) to allow sensing of a pitch tilt angle (see paragraph 42) Powell does not expressly disclose wherein the sensor includes an accelerometer. Dao discloses wherein the sensor includes an accelerometer (see paragraph 49 note that tilt angle is determined with an accelerometer). One of ordinary skill in the art could have easily substituted the accelerometer of Dao for the tilt sensor of Powell and the results would have been the same merely the tilt angle is computed by a different type of sensor and therefore predictable. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Powell and Dao.
Claim(s) 16 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Powell US 2023/0230210 in view of Burenius US 2016/0057385
Re claim 16 Powell discloses the camera or device incorporating the camera also may include a tilt sensor (e.g. an inertial motion sensor, rotational encoder incorporated into the pivot joint, etc.) to allow sensing of a pitch tilt angle (see paragraph 42) Powell does not expressly disclose wherein the sensor includes a directional microphone. Burenius discloses wherein the sensor includes a directional microphone (see paragraph 13 and 16 note that the microphone may be used to control the camera relative to participated also see paragraph 24 “an audio processor proximate microphone array 118 to determine audio angle of arrival and distance from a sound source (as discussed above)”. The motivation to combine is to switch to a camera or adjust its angle to capture a participant (see paragraph 47 and 13). One of ordinary ordinary skill in the art could have easily combined Powell and Bruenius to use the microphone to adjust and determine the camera relative to a participant of interest. Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Powell and Bruenius to reach the aforementioned advantage.
Re claim 35 Powell discloses all of the elements of claim 1 Powell does not expressly disclose wherein the videoconferencing system is a multi-camera videoconferencing system including a plurality of cameras, the camera being included in the plurality of cameras. Bruenius discloses wherein the videoconferencing system is a multi-camera videoconferencing system including a plurality of cameras, the camera being included in the plurality of cameras (see paragraph 47 note Bruenius discloses switching between a plurality of cameras based on who is speaking) The motivation to combine is to switch to a camera or adjust its angle to capture a participant (see paragraph 47 and 13). Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Powell and Bruenius to reach the aforementioned advantage.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Powell US 2023/0230210 and “Powell ‘457” US 20220366547 A1 hereinafter “Powell ‘457 in view of Martin US 2004/0088732.
Re claim 19 Powell discloses wherein the image transformation indicates one or more image adjustments, on a pixel-by-pixel basis, (see paragraph 30 note that adjustments are made on a pixel to pixel basis based on the tilt angle)
Re claim 19 Powell discloses wherein the image transformation indicates one or more image adjustments, on a pixel-by-pixel basis, based on a tilt angle (see paragraph 30 note that adjustments are made on a pixel to pixel basis based on the tilt angle)
Powell does not expressly disclose dependent on a difference between a first camera pan angle associated with the actual camera perspective and a second camera pan angle associated with the target camera perspective. Powell ‘547 discloses dependent on a difference between a first camera angle associated with the actual camera perspective and a second camera angle associated with the target camera perspective (see paragraph 69 34 and note that the amount of correction it based on a difference between the tilt angle e.g. 17 degree and a target tilt angle no tilt i.e. zero degrees. Partial tilt correction may also be applied or full correction for 17 degrees) The motivation to combine is “In such a correction, the projection mapping may correct for distortion while maintaining a rectilinear-like appearance. That is, vertical-line objects may appear straight while curvature may be relatively small, locally, for horizontal-line objects near the top or bottom of the corrected image. Additionally, in some examples, the projection mapping further may be based on a sensed camera tilt angle. As such, the mapping may adjust for a vertical angle bias”. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Powell and Powell ‘547 to reach the aforementioned advantage.
Powell and Powell ‘547 do not expressly disclose using a pan angle. Martin discloses correcting and image perspective for pan angle (see paragraph 24 note that that pan angle may be corrected as well as tilt angle). The motivation to combine is to correct for both pan and tilt (see paragraph 24). One of ordinary skill in the art could have easily modified Powell and Powell ‘547 with the teachings of Martin to correct for both Pan and tilt. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Martin with Powell and Powell ‘547 to reach the aforementioned advantage.
Claim(s) 20 and 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Powell US 2023/0230210 in view of Powell US 2022/0366547 hereinafter “Powell ‘457”
Re claim 20 Powell discloses wherein the image transformation indicates one or more image adjustments, on a pixel-by-pixel basis, based on a tilt angle (see paragraph 30 note that adjustments are made on a pixel to pixel basis based on the tilt angle)
Powell does not expressly disclose dependent on a difference between a first camera tilt angle associated with the actual camera perspective and a second camera tilt angle associated with the target camera perspective. Powell ‘547 discloses dependent on a difference between a first camera tilt angle associated with the actual camera perspective and a second camera tilt angle associated with the target camera perspective (see paragraph 69 34 note that the amount of correction it based on a difference between the tilt angle e.g. 17 degree and a target tilt angle no tilt i.e. zero degrees. Partial tilt correction may also be applied or full correction for 17 degrees) The motivation to combine is “In such a correction, the projection mapping may correct for distortion while maintaining a rectilinear-like appearance. That is, vertical-line objects may appear straight while curvature may be relatively small, locally, for horizontal-line objects near the top or bottom of the corrected image. Additionally, in some examples, the projection mapping further may be based on a sensed camera tilt angle. As such, the mapping may adjust for a vertical angle bias”. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Powell and Powell ‘547 to reach the aforementioned advantage.
Re claim 32 Powel does not expressly disclose “wherein application of the at least one image transformation is accomplished using an image warp mesh indicating, for a plurality of pixel coordinates of the image warp mesh, one or more transformations to apply relative to pixel coordinates of the overview video stream.” Powell ‘457 discloses wherein application of the at least one image transformation is accomplished using an image warp mesh indicating, for a plurality of pixel coordinates of the image warp mesh, one or more transformations to apply relative to pixel coordinates of the overview video stream (see paragraph 76 31 and 52 note that a mesh file is used to correct images on a pixel by pixel basis ). The motivation to combine is “In such a correction, the projection mapping may correct for distortion while maintaining a rectilinear-like appearance. That is, vertical-line objects may appear straight while curvature may be relatively small, locally, for horizontal-line objects near the top or bottom of the corrected image. Additionally, in some examples, the projection mapping further may be based on a sensed camera tilt angle. As such, the mapping may adjust for a vertical angle bias”. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Powell and Powell ‘547 to reach the aforementioned advantage.
Claim(s) 26, 28 and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Anderson US 2022/0334638 in view of Isikdogen US 2023/0410266.
Re claim 26 Anderson discloses all of the elements of claim 1. Anderson does not expressly discloses provide at least one trained neural network configured to receive the at least one test frame and an indicator of the target camera perspective as inputs and output the at least one image transformation. Isikdogen discloses t least one trained neural network configured to receive the at least one test frame and an indicator of the target camera perspective as inputs and output the at least one image transformation (see paragraph 12 and figure 1 note that a neural network inputs a target angle and input image and outputs a vector field used to warp the image for gaze correction). The motivation to combine is “Moreover, the techniques may be used in a live application that runs in real-time on CPU, making the techniques a feasible solution for a wide range of devices”. One of ordinary skill in the art could have easily used the techniques of Isikdogen to perform the warping of Anderson to achieve the aforementioned advantage. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Isikdogen and Anderson to reach the aforementioned advantage.
Re claim 28 Anderson discloses provide at least one trained neural network configured to receive the at least one test frame, determine the at least one region of interest represented in the test frame, determine the one or more indicators of actual camera perspective, (see paragraph 37 note that the network determines the location of the eyes and the gaze direction with respect to the camera)
Anderson does not expressly disclose provide at least one trained neural network configured to receive an indicator of the target camera perspective as input, determine the at least one image transformation, and output the at least one image warped primary video stream
Isikdogen discloses wherein the at least one processor is further programmed to provide at least one trained neural network configured to receive the at least one test frame and an indicator of the target camera perspective as inputs (see figure 1 and paragraph 12 note that the CNN takes in a target angle and an input image), determine the one or more indicators of actual camera perspective (see paragraph 12 note that an input gaze direction relative to the camera may be determined), determine the at least one image transformation, and output the at least one image warped primary video stream (see paragraph 12 and figure 1 note that a neural network inputs a target angle and input image and outputs a vector field used to warp the image for gaze correction). The motivation to combine is “Moreover, the techniques may be used in a live application that runs in real-time on CPU, making the techniques a feasible solution for a wide range of devices”. One of ordinary skill in the art could have easily used the techniques of Isikdogen to perform the warping of Anderson to achieve the aforementioned advantage. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Isikdogen and Anderson to reach the aforementioned advantage.
Re claim 29 Anderson discloses wherein the at least one processor is further programmed to provide at least one trained neural network configured to receive the at least one test frame as an input, determine the at least one region of interest represented in the test frame, determine the one or more indicators of actual camera perspective, (see paragraph 37 note that the network determines the location of the eyes and the gaze direction with respect to the camera ) determine the target camera perspective, (see paragraph 44 and 41 note that the gaze management circuitry determines whether or not to perform the correction to correct the eyes looking at the camera this effectively set the target perspective)
Anderson does not expressly disclose provide at least one trained neural network configured to determine the at least one image transformation, and output the at least one image warped primary video stream.
Isikdogen discloses provide at least one trained neural network configured to determine the at least one image transformation, and output the at least one image warped primary video stream. (see paragraph 12 and figure 1 note that a neural network inputs a target angle and input image and outputs a vector field used to warp the image for gaze correction). The motivation to combine is “Moreover, the techniques may be used in a live application that runs in real-time on CPU, making the techniques a feasible solution for a wide range of devices”. One of ordinary skill in the art could have easily used the techniques of Isikdogen to perform the warping of Anderson to achieve the aforementioned advantage. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Isikdogen and Anderson to reach the aforementioned advantage.
Cited Art
The following is a recitation of art considered relevant but not used in a rejection above:
VYAS US 20230254437 discloses: Image de-skewing can include acquiring pixel coordinates of an image captured by a camera adjoining an electronic display device. The pixel coordinates can be mapped to de-skewing coordinates using a predetermined de-skewing transformer that maps the pixel coordinates to de-skewing coordinates. A de-skewed image can be generated based on the de-skewing coordinates such that the de-skewed image is perpendicular to a principal axis extending from a predetermined location of an apparent camera on the electronic display device. (see abstract).
Powell US 20240112315 A1 discloses However, some camera placements and arrangements of objects in the imaged scene may pose challenges to distortion correction. For instance, a videoconferencing camera may be placed at one end of a room, for example high on a wall above, or attached to the top of, a videoconference display, and tilted downwards to capture subjects seated around a conference table. In this example, objects at a far end of the conference table opposite the camera may be located at a relatively far distance from the camera near the center of the raw image, while objects at the near end of the conference table adjacent to the camera may be located relatively close to the camera near the edges of the raw image. As objects close to the camera may be located at extreme viewing angles, this may lead to foreshortening effects and/or trapezoidal distortion for these objects (e.g., a person, table, etc.). At the same time, objects farther from the camera may appear small compared to other subjects. As a result, using a wide-angle or an ultra-wide-angle lens in a conference room setting, for example, people in an image standing close to the camera and/or near the sides of the FOV may appear to be warped, among other distortions. Such distortions may be further apparent when performing pan, tilt, and/or zoom functions that are software-implemented (e.g., as opposed to mechanically, or optically, implemented). See paragraph 37
Ranjan US 20180181809 A1 discloses [0097] Also, as shown in operation 808, a head pose estimation is performed to determine yaw, pitch, and roll values for the image of the face relative to a camera plane. In addition, as shown in operation 810, normalization is performed on the head of the subject to correct for the roll value and to warp the image via perspective projection to be from the viewpoint of a normalized camera that is looking directly at the eye. Furthermore, as shown in operation 812, yaw and pitch values of the head with respect to the normalized camera are determined for the corrected head of the subject. Further still, as shown in operation 814, the image of the eye and the yaw rotation and pitch rotation of the head are provided to a CNN. Also, as shown in operation 816, the CNN determines a yaw rotation and pitch rotation of the eye.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN T MOTSINGER whose telephone number is (571)270-1237. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN T MOTSINGER/ Primary Examiner, Art Unit 2673
Prosecution Insights