Detailed Action
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Acknowledgements
2. Applicant’s arguments, filed on 12/17/2025, are acknowledged. Amended claims 1-2, 7-10, 15, 20, 25-29, 44, 46-50 and 53 and cancelled claims 6, 16, 34, and (51; previously cancelled) 52 are acknowledged.
Claims 1-5, 7-15, 17-33, 35-50, and 53 remain pending and have been examined.
Response to Arguments
3. Applicant’s arguments, see pg. 14-15 of 22, filed on 12/17/2025, with respect to the amended claims 1 and 50 under 35 U.S.C. 102 (a)(1) have been fully considered
and are persuasive. Therefore, the rejection has been withdrawn.
However, upon further consideration, a new ground(s) of rejection is made in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak).
4. Applicant argues/remarks:
Saa fails to disclose at least the following features of the Applicant’s currently amended independent claims 1 and 50:
for each of two or more pixels of the frame:
isolating translational motion from rotational motion from said encoding to extract motion data for the pixel from the encoded stable field of view; and
mapping the pixel to a corresponding pixel in the one-, two-, or more-dimensional projection, based on the extracted motion data for the pixel.
5. Response:
Pathak teaches that frame to frame motion of a camera can be defined in 5 Degrees of Freedom (3 for rotation, 2 for translation). Pg. 3, column 1-2, further teach the obtaining of image point moving translationally describing the path of pixels on an image for a camera undergoing translation; and image points of rotational movement wherein the optical flow of pure translation and rotation are calculated using the Farneback algorithm. As such, wherein flow vectors are projected on a sphere as stated on pg. 3 column 2 (with regards to figures 2 and 3).
Pathak, as stated on pg. 1-column 2, thus teaches image stabilization by estimating the rotation of a spherical video thus to produce a non-warped image using dense optical field information.
Information Disclosure Statement
6. The information disclosure statement (IDS) submitted on 02/11/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
7. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 1-2, 6-9, 15-16, 19, 44-50, and 53 are rejected under 35 U.S.C. 103 as
being unpatentable over by Saa-Garriga et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak).
9. Regarding claim 1, a computer-implemented method for stabilising motion data (…wherein Saa, in [0014], teaches obtaining sensor data generated as a result of sensing shaking; [0077-79] further teaches a computer program to perform image correction caused by shaking…), the method comprising:
receiving data obtained by one or more visual sensors of a first object (…wherein [0005] teaches obtaining a plurality of motion vectors regarding a 360-degree image; in which at least one moving object is detected in the 360-degree image…), wherein
the first object has a variable position and orientation relative to one or more second objects in an environment of the first object (…wherein [0063] teaches a motion vector relative to an object that moves less in a scene; wherein the “scene” relative to the object can be viewed as a second object…), wherein
the data comprises
a frame comprising a plurality of pixels (…wherein [0051] teaches a format of the 360-degree image being in pixels forming frames…);
establishing, from said data, a rotationally stable field of view for the one or more visual sensors using one or more techniques (…wherein [0005] teaches shake distortion correction based on 3D rotation information. Further, [0043] teaches the correcting of distortion of a 360-degree image by combining sensor data with 3D rotation…);
encoding the stable field of view based on at least a one-, two- or more- dimensional projection (…wherein [0077] teaches that an image processing apparatus may render and display a corrected 360-degree image or may encode and store the image…).
Saa doesn’t further teach wherein
for each of two or more pixels of the frame:
isolating translational motion from rotational motion from said encoding to extract motion data for the pixel from the encoded stable field of view (…Pathak teaches that frame to frame motion of a camera can be defined in 5 Degrees of Freedom (3 for rotation, 2 for translation). Pg. 3, column 1-2, further teaches the obtaining of image point moving translationally describing the path of pixels on an image for a camera undergoing translation; and image points of rotational movement wherein the optical flow of pure translation and rotation are calculated using the Farneback algorithm…); and
mapping the pixel to a corresponding pixel in the one-, two-, or more- dimensional projection, based on the extracted motion data for the pixel (…wherein flow vectors are projected on a sphere (Fig. 3); as stated on pg. 3-column 2.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that finding the rotation between two spherical image so to derotate one image to a common orientation based on optical flow in order to separate translational and rotational components which can yield motion parameters of the camera, so to present an un-warped image in output…).
10. Regarding claim 2, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein said
one or more techniques comprise:
algorithms configured to create a rotationally stabilised omnidirectional field of view from
the said data based on the orientation of said one or more sources (…wherein the flowcharts
of figures 2 and 3 may be viewed as algorithms which are used to obtain a 360 degree
image; wherein [0182] teaches that such software elements of programming may be
implemented with any language (e.g., C++, assembler language) with various algorithms being implemented with any combination of data structure, objects, processes, routines
or other programming elements…).
11. Regarding claim 7, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above),wherein said one-two-or more dimensional projection or a cylindrical projection (…wherein [0051] teaches according to unit sphere representation, pixels forming frames of the 360-degree image may be indexed to a three-dimensional (3D) coordinate system defining locations of respective pixels on a surface of a virtual sphere…).
12. Regarding claim 8, Saa in view of Pathak teaches the method of claim 7 (see claim 7 above), wherein the spherical projection of the object moves with said one or more visual sensors (…wherein Saa, in [0062], as part of explaining global rotation teaches that when a 360-degree image is captured in a vehicle that moves, global rotation may occur in the background due to the rotation of the vehicle and may occur in every part of the vehicle shown in the background and the foreground due to the rotation of the camera itself…).
13. Regarding claim 9, Saa in view of Pathak teaches the method of claim 7 (see claim 7 above), wherein the stable fields of view is at least partially non-stabilised in relation to at least one of the second objects (…while Saa, in [0062], teaches global rotation, [0063] teaches motion vectors related to motion of an object within scene. [0065] teaches the performance of filtering to remove a motion vector. Therefore, it may be said that a fully stabilized field of view requires both global rotation and motion vectors to be corrected…).
14. Regarding claim 15, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein said one-, two-, or more-dimensional projection is an equi-area projection and/or locally Cartesian projection (…wherein Saa, in [0052], teaches 2D equivalent representation such as a cube map projection or an equi-rectangular projection, regarding the format in which a 360-degree image is stored...).
15. Regarding claim 19, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein the method is implemented on one or more processors associated with at least one of:
a central processing unit (…wherein [0182] teaches that Saa’s disclosure may
employ CPUs…).
16. Regarding claim 44, Saa Pathak teaches the method of claim 1 (see claim 1 above), wherein said one or more visual sensors comprise:
at least one camera, sensor, or device suitable for receiving external data direction or indirectly (…wherein Saa, in [0157], teaches a data obtainer which may communicate with an eternal device to obtain learning data related to a 360 degree image…).
17. Regarding claim 45, Saa in view Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
receiving simulated data in relation to one or more simulations, wherein the simulated
data are used to establish the stable field of view by means of insertion or superposition of the
simulated data to said data (…wherein Saa, in [0075], teaches an embodiment based on
artificial intelligence (AI), where machine learning mechanism is used to imitate actions
of living things and obtain sensor rotation translation by using motion vectors as input
data; [0157], further teaches a data obtainer which may communicate with an eternal
device to obtain learning data related to a 360 degree image. Further, [0160] teaches a
model learner may learn standards regarding whether to determine the 3D rotation
information from the motion vectors, by using some information from the 360-degree
image in layers of the learning model network. Wherein the data received for learning can
be viewed as an element of simulation data, the use of information from the 360 degree
image in layers can be viewed as superimposed information…).
18. Regarding claim 46, Saa in view Pathak teaches the method of claim 1(see claim 1 above), wherein the said data comprise:
simulated data corresponding to said one or more visual sensors for establishing the stable field of view with said data (…wherein Saa, in [0157], teaches a data obtainer which may communicate with an eternal device to obtain learning data related to a 360 degree image (simulation data); [0160], teaches a model learner which may learn standards regarding whether to determine the 3D rotation information from the motion vectors, by using some information from the 360-degree image in layers of the learning model network. As limited in claim 1, rotational information is used as part of generating distortion corrected image…).
19. Regarding claim 47, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
applying one or more machine learning (ML) models to classify an object in said data
based on said one-, two-, or more- dimensional projection (…wherein Saa, in [0075], teaches machine learning system may be used that is the same as a learning network model that trains with regard to patterns of motion vectors in a frame having predetermined rotation; thus the patterns of motion vectors can be viewed objectively…),
wherein said one or more ML models configured to recognise the object representative
of said encoding associated with the stable field of view (…further, [0059] teaches that motion
vector may be generally generated and stored during an existing image encoding
process…).
20. Regarding claim 48, Saa in view of Pathak teaches the method of claim 47 (see claim 47 above), wherein said one or more ML models are trained using data annotated with one or more objects wherein the annotated data is transformed using said one-, two-, or more- dimensional projection for training the ML models (…Saa in [0151] teaches a data learner may learn standards for obtaining 3D rotation information from motion vectors regarding a 360-degree image and a data recognizer may determine the 3D rotation information from the motion vectors regarding the 360-degree image, based on the standards that are learned by the data learner…).
21. Regarding claim 49, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
generating a labelled output dataset for the training a machine learning model from a
dataset of labelled sensor inputs (…Saa, in [0073], teaches the use of a learning network
model to obtaining of rotational information based on motion vectors. Further, [0075]
teaches the conversion of sensory information into a format corresponding to motor
system requirements; wherein machine learning mechanism are used to imitate natural
actions and obtain sensor rotation/translation by using motion vectors as input data…),
wherein the machine learning model is configured to operate on said data stored and
encoded by said one-, two-, or more- dimensional projection (…Saa, in [0076], teaches step S240, wherein an image processing apparatus corrects distortion of a 360-degree image caused by shaking, based on the obtained 360-degree rotation information…).
22. Regarding claim 50, an apparatus for stabilising motion data (…wherein Saa, in [0014], teaches obtaining sensor data generated as a result of sensing shaking; [0077-79] further teaches a computer program to perform image correction caused by shaking…), comprising:
an interface for receiving data from one or more visual sensors of a first object (…wherein [0005] teaches obtaining a plurality of motion vectors regarding a 360-degree image; in which at least one moving object is detected in the 360-degree image…), wherein
the first object has a variable position and orientation relative to one or more second objects in an environment of the first object (…wherein [0063] teaches a motion vector relative to an object that moves less in a scene; wherein the “scene” relative to the object can be viewed as a second object…), wherein
the data comprises a frame comprising a plurality of pixels (…wherein [0051] teaches a format of the 360-degree image being in pixels forming frames…);
one or more integrated circuits (…wherein Saa teaches figures 17 and 18 which may
be viewed as integrated circuits…) configured to:
establish, from said data, a rotationally stable field of view for the one or more visual sensors using one or more techniques (…wherein [0005] teaches shake distortion correction based on 3D rotation information. Further, [0043] teaches the correcting of distortion of a 360-degree image by combining sensor data with 3D rotation…);
encode the stable field of view based on a one-, two-, or more- dimensional projection (…wherein [0077] teaches that an image processing apparatus may render and display a corrected 360-degree image or may encode and store the image…).
Saa doesn’t further teach wherein
for each of two or more pixels of the frame:
isolating translational motion from rotational motion from said encoding to extract motion data for the pixel from the encoded stable field of view (…Pathak teaches that frame to frame motion of a camera can be defined in 5 Degrees of Freedom (3 for rotation, 2 for translation). Pg. 3, column 1-2, further teaches the obtaining of image point moving translationally describing the path of pixels on an image for a camera undergoing translation; and image points of rotational movement wherein the optical flow of pure translation and rotation are calculated using the Farneback algorithm…); and
mapping the pixel to a corresponding pixel in the one-, two-, or more- dimensional projection, based on the extracted motion data for the pixel (…wherein flow vectors are projected on a sphere (Fig. 3); as stated on pg. 3-column 2.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that finding the rotation between two spherical image so to derotate one image to a common orientation based on optical flow in order to separate translational and rotational components which can yield motion parameters of the camera, so to present an un-warped image in output…).
23. Regarding claim 53, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
mapping two or more pixels of the frame to a same corresponding pixel in the one, two
or more dimensional projection, based on the extracted stable motion data for the two or more
pixels (…wherein two or more dimensional projection based on extracted stable motion data based on image frames (thus corresponding pixels) is addressed in claim 1;
Saa, in [0051], further teaches pixels forming frames of a 360-degree image may be indexed to a 3D coordinate system defining locations of respective pixels…).
24. Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et al.
(US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further view of Soman et al. (US 2022/0294987 A1; further referred to as Soman).
25. Regarding claim 3, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above). Saa in view Pathak, however, does not further teach the method of claim 1 wherein said one or more techniques further comprise algorithms configured to correct rolling shutter from said data (…however, Soman teaches an image stabilization process which may compensate for rolling shutter distortions during image capture; wherein the process also includes compensating for effects of movements of an imaging camera device due to rotation and translation of the device (as taught in [0047]).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that image stabilization can be enhanced
through a means of compensating for rolling shutter distortions…).
26. Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et
al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further of view of Wissenbach et al. (US 2018/0332226 A1; further referred to as Wissenbach).
27. Regarding claim 4, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above). Saa does not further specify the method of claim 1 wherein said one or more techniques process said data by iteratively adding received data in a continuous manner to establish the stable field of view (…however, Wissenbach teaches an omnidirectional camera of image stabilization and reorientation which utilizes image stabilization iterations in combination with data interpolation and/or derivation ([0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that iterations through accumulated data can
be performed so to generate image stabilized spherical video files…).
28. Regarding claim 5, Saa in view Pathak and further view of Wissenbach teaches the method of claim 4 (see claim 4 above), wherein the processed data is at least partially stored in memory (…wherein Saa, in [0044], teaches a processor configured to execute one or more instructions stored in memory, wherein the processor is configured to: obtain a plurality of motion vectors regarding a 360-degree image…).
Saa does not further teach the method of claim 4:
wherein the received data is processed in real-time without storing said data in memory
(…however, Wissenbach, in [0047], teaches the capturing of live video by a group
cameras in one or more consecutive frames.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a group of cameras can be coordinated
to provide a multidirectional view of an environment with the need to be observed in live
mode and thus be processed accordingly…).
29. Claims 10-14, 20-23, and 25-26 are rejected under 35 U.S.C. 103 as being
unpatentable over Saa-Garriga et al. (US 2021/0142452 A1; further referred to as Saa)
in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further view of Calabretta (Mapping on the HEALPix grid).
30. Regarding claim 10, Saa in view of Pathak teaches the method of claim 1(see claim 1 above). Though Saa, in [0052], teaches cube map projection and equi-rectangular projection, Saa does not further teach the method of claim 1 wherein said one-, two-, or more- dimensional projection comprise:
a Hierarchical Equal Area isoLatitude Pixelization (HEALPix) projection (…however, Calabretta teaches HEALPix projection as a simple way of storing HEALPix data on a two- dimensional square grid as used in conventional imaging and mapping (pg. 2-paragraph 3).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image...).
31. Regarding claim 11, Saa in view of Pathak and further view of Calabretta teaches the method of claim 10 (see claim 10 above),wherein the HEALPix projection applies a HEALPix double pixelization derivative (…Calabretta teaches an extension to the HEALPix pixelisation (double- pixelization) wherein additional pixels are added to a HEALPix grid, thus increasing the total number of pixels being mapped (pg. 4-paragraph 3).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix double-pixelization, as taught
by Calabretta, can further enhance the formatting and storing of a 360-degree image...).
32. Regarding claim 12, the method of claim 10 (see claim 10 above), further comprising:
applying a 2^n HEALPix n_side parameters with the HEALPix projection (…wherein Calabretta teaches a scheme for distributing 12N^2 points as uniformly as possible over a surface of a unit sphere.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
Image…).
33. Regarding claim 13, Saa in view of Pathak and further view of Calabretta teaches the method of claim 10 (see claim 10 above), wherein said motion data is extracted using an algorithm for estimating motion (…wherein Saa, in [0114], teaches the use of an object detection algorithm to detect at least one moving object from the 360 degree image…),
wherein the algorithm is configured with respect to properties of equal pixel area and
locally Cartesian nature (…wherein Calabretta, on pg. 6-paragraph 7, teaches HEALPix
projections denoted in FITS with an algorithm code; it would have been obvious to one of
ordinary skill in the art before the effective filing date of the claimed invention that the
grid of a HEALPix pixelisation or double-pixelation (see Fig. 3 and Fig. 4; Calabretta) can
further be represented in a coordinate system of cartesian mapping with regards to
smaller defined areas
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image…).
34. Regarding claim 14, Saa in view of Pathak and further view of Calabretta teaches the method of claim 10 (see claim 10 above), wherein said motion data is extracted using optical flow (…Saa, in [0057], teaches that motion vector may represent displacement information of a particular area of an image with regards to a reference frame and a current frame; this may be viewed as optical flow wherein motion may be apparent by movement of pixels between two frames…).
35. Regarding claim 20, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above) wherein the one-, two-, or more-dimensional projection is a spherical projection. Saa does not teach the method further comprising:
extracting orthogonal bands in relation to said one or more data structure associated
with the spherical projection, wherein the orthogonal bands are about the identifiable cartesian axes of the spherical projection (…Calabretta teaches members of HEALPix projection wherein, e.g. in Fig 1, extraction of orthogonal bands can correspond to the rescaled projections of the members as depicted in Fig. 1. Therein, it is evident that the members of the projection (of H = 1-4) can be further defined by a cartesian axes (x,y).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image...).
36. Regarding claim 21, Saa in view of Pathak and further view of Calabretta teaches the method claim 20 (see claim 20 above), wherein the spherical projection is HEALPix (…Calabretta teaches HEALPix projection as a simple way of storing HEALPix data on a two-dimensional square grid as used in conventional imaging and mapping (pg. 2-paragraph 3).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image...).
37. Regarding claim 22, Saa in view of Pathak and further view of Calabretta teaches the method of claim 20 (see claim 20 above), wherein the spherical projection applies a double pixelization (…wherein Calabretta teaches double-pixelisation on the HEALPix projection (pg. 4 (3.1 HEALPix double pixelisation), as depicted in Fig. 4…).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image...).
38. Regarding claim 23, Saa in view of Pathak and further view of Calabretta teaches the method of claim 20 (see claim 20 above), further comprising:
applying spatial filtering on the spherical projection by use of the orthogonal bands
(…wherein, Saa teaches in [0020] teaches the removing of motion vectors of a preset
area through filtering; Calabretta teaches members of HEALPix projection wherein, e.g.
in Fig 1, extraction of orthogonal bands can correspond to the rescaled projections of
the members as depicted in Fig. 1. Therein, it is evident that the members of the
projection (of H = 1-4) can be further defined by a cartesian axes (x,y).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that HEALPix projection, as taught by
Calabretta, could have been implemented in the formatting and storing of a 360-degree
image and thus be used for filtering purposes of motion vectors...).
39. Regarding claim 25, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein the one-, two-, or more-dimensional projection is a cylindrical projection, the method further comprising:
extracting orthogonal bands in relation to said one or more data structure associated
with the cylindrical projection (…wherein Calabretta, on pg. 1-Fig. 1, denotes that the
HEALPix class of projections to reveal the underlying cylindrical equal-area projection in
the equatorial region…),
wherein the orthogonal bands are about the identifiable Cartesian axes of the cylindrical
projection in a manner to capture a direction based on vertical strips of said data (…wherein
the depiction of Fig. 1 shows members (orthogonal bands) of the projection which are
depicted in a cartesian grid.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a method of mapping as taught by
Calabretta can be used to implement to perform filtering of motion vectors as taught by
Saa…).
40. Regarding claim 26, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein the one-, two-, or more- dimensional projection is a spherical projection, the method further comprising:
extracting orthogonal bands in relation to said one or more data structures associated
with the spherical projection (…wherein Calabretta teaches members of HEALPix projection
wherein, e.g. in Fig 1, extraction of orthogonal bands can correspond to the rescaled
projections of the members as depicted in Fig. 1…),
wherein the extracted orthogonal bands are adapted to be applied with an algorithm
associated with a projection (…Calabretta, on pg. 6-paragraph 6, teaches HEALPix
projections being denoted with an algorithm code…),
wherein the extracted orthogonal bands are used as an encoding on said one or more
data structures (…wherein Calabretta, on pg. 6-paragraph 6, teaches HEALPix projections
will be denoted1 in FITS (flexible image transport system.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that the teachings of Saa could have been
implemented in a mapping and data structure method of a sphere as taught by
Calabretta…).
41. Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et al.
(US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further view of Taylor (The basics of FPGA mathematics EE Times (Year: 2012)).
42. Regarding claim 17, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein the method is implemented on a field-programmable gate array using a fixed-point implementation (…wherein Saa teaches the use of a FPGA for a software and hardware component to perform function of its specification; Taylor further teaches the representation of fixed-point number system within a design (algorithm) used in FPGA.
Therefore it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that fixed-point number system implemented
in a FPGA could have been implemented wherein fixed-point representation maintains a
decimal point within a fixed position which simplifies arithmetic operations…).
43. Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et al.
(US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further view of Harris (Mixed-Precision Programming with CUDA 8).
44. Regarding claim 18, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above). wherein
the method is implemented on a vision accelerator unit using 16-bit floating-point arithmetic (…wherein Saa, in [0044], teaches the use FPGA or computer vision accelerator chip
implementation. However, Saa does not specify the use of 16-bit floating-point arithmetic
of the FPGA or computer vision accelerator chip.
However, Harris teaches the NVIDIA Tesla P100 (enabled by Pascal architecture)
which can perform FP-16 arithmetic for deep neural network architectures; wherein, as taught on pg. 3-paragraph 3, Harris teaches that a 16-bit floating point arithmetic suffices
for deep neural network for GPU related applications.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a computer vision accelerator chip (as
taught by Saa) may be implemented by devices as taught by thereby providing single-
precision high performance computing…).
45. Claims 24 and 27-28 are rejected under 35 U.S.C. 103 as being unpatentable over Saa-
Garriga et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and Calabretta (Mapping on the HEALPix grid) and further view of Kim et al. (US 2021/0366081 A1; further referred to as Kim).
46. Regarding claim 24, Saa in view of Pathak and Calabretta teaches the method of claim 20 (see claim 20 above). The combined reference does not teach the method further comprising:
performing a 2D convolution on the spherical projection by generating 1D convolution
around each of the orthogonal bands to improve said performance of the 2D convolution (…however, Kim teaches an image processing method using a line unit operation,
including a first and second convolution operators; wherein [0115] teaches a first 1D
convolution operator to perform a 1D convolution operation (wherewith as taught in
[0101], a first convolution operator may generate a feature map by performing); [0162]
teaches a 2D convolution operator configured to perform a 2D convolution operation
(wherewith as taught in [0103] the second convolution operator may perform a
convolution operation based on a feature map output in the 2D form.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that different dimensional convolutions can
be applied to an image with different parts of the image undergoing a different convolution…).
47. Regarding claim 27, Saa in view of Pathak and Calabretta teaches the method of claim 20 (see claim 20 above), wherein the orthogonal bands applied simultaneously to generate convolutions in a parallel manner based on the projection (…wherein Calabretta teaches spherical projections broken up into members, Kim further teaches convolutional operations that may be performed. Further, Kim, in [0184], teaches components included in the image processing apparatus 10 may operate concurrently or in parallel to process an image; wherein apparatus 10 includes controller 200 (as depicted in Fig. 1), controller 200 includes a first and second convolution operators (as depicted in Fig. 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that parallel processing methods with
regards to convolution, as taught by Kim, could have been implemented with the
combined teachings of Saa in view of Calabretta thus implement a fast processing
pipeline structure…).
48. Regarding claim 28, Saa in view of Pathak and Calabretta and further view of Kim teaches the method of claim 27 (see claim 27 above), wherein each of the orthogonal bands are segmented for parallel processing to generate convolutions associated with the projection (…wherein Calabretta teaches spherical projections broken up into members, Kim further teaches convolutional operations that may be performed. Further, Kim, in [0184], teaches components included in the image processing apparatus 10 may operate concurrently or in parallel to process an image; wherein apparatus 10 includes controller 200 (as depicted in Fig. 1), controller 200 includes a first and second convolution operators (as depicted in Fig. 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that parallel processing methods with regards to convolution, as taught by Kim, could have been implemented with the
combined teachings of Saa in view of Calabretta thus implement a fast processing
pipeline structure…).
49. Claims 29-32 and 42-43 are rejected under 35 U.S.C. 103 as being unpatentable over
Saa-Garriga et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and further view of Cote et al. (US 2015/0296193 A1; further referred to as Cote).
50. Regarding claim 29, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above). Saa doesn’t further teach the method wherein data from one or more visual sensors are demosaiced (…however, Cote, in [0008], teaches converting raw pixel data into RGB data using demosaic.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that raw pixel data can be used so to
interpolate missing color values of each pixel…).
51. Regarding claim 30, Saa in view of Pathak and Cote teaches the method of claim 29 (see claim 29 above). Saa doesn’t further teach the method wherein the data is RGB data corresponding to visual information (…however, Cote, in [0008], teaches converting raw pixel data into RGB data using demosaic.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that raw pixel data can be used so to
interpolate missing color values of each pixel…).
52. Regarding claim 31, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above). The combined reference doesn’t teach the method further comprising:
applying pixel binning and down sampling to create a rotationally stabilized
omnidirectional field of view (…however, Cote, in [0523] teaches downsampling RGB image
data and further in [0728] teaches binning to produce binned raw image pixel.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that downsampling to improve signal-to-
noise ratio and further produce thumbnails by downsampling…).
53. Regarding claim 32, Saa in view of Pathak and Cote teaches the method of claim 31 (see claim 31 above), wherein the pixel binning is configured for debayering said data by separately accumulating three colour channels (…wherein Cote in [0730] teaches that with regard to binned data, 2x2 pixel data may form a bayer pattern and may be determined by
averaging the values of the pixels from a full resolution raw image data.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that binning, as taught by Cote, is
implemented as an effective way to achieve higher sensitivity of light sensing…).
54. Regarding claim 42, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), wherein said data comprises non-RGB data, wherein the non-RGB data are associated with non-colour information (…wherein Saa, in [0151], teaches a data learner which may learn standards for obtaining 3D rotation information from motion vectors regarding a 360 degree image …).
Saa, in [0014], generally teaches image data capturing with a device. However, Saa
does not specify the wherein said data comprises RGB data (…however, Cote teaches RGB
formatted image data processing in [0300].
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that the image capturing device, as taught by Saa, could have been implemented with a device that processes RGB data as taught by
Cote, so to employ color filtering for better color defined pixels…).
55. Regarding claim 43, Saa in view of Pathak and Cote teaches the method of claim 42 (see claim 42 above), wherein the non-RGB data comprise:
data from other external sources (…wherein Saa, in [0157], teaches a data
obtainer which may communicate with an external device to obtain learning data related
to a 360 degree image…).
56. Claims 33, 35, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and Baxter et al. (US 2004/0095492 A1; further referred to as Baxter).
57. Regarding claim 33, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
applying heterogeneous sensing to the stable field of view, wherein the stable field of
view is omnidirectionally established based on one or more techniques wherein the heterogeneous sensing comprises encoding at least part of the stable field of view at a higher spatial resolution (…wherein Saa teaches a stable field of view (namely, a distortion corrected 360-degree image and its encoding), Baxter teaches in an imaging system, as taught in [0009], including a VASI (higher resolution) subsystem with the capability of retaining the highest possible spatial resolution on regions of interest that are important to the overall system.
Therefore, it would have been obvious to one skilled in the art before the effective
filing date of the claimed invention that a variable acuity superpixel imager (VASI) could
have been implemented as a heterogeneous method of sensing received light by an
imager with different resolution, so to provide better definition to particular areas of an
image…).
58. Regarding claim 35, Saa in view of Pathak and Baxter teaches the method of claim 33 (see claim 33 above), further comprising:
sampling over regions of the stable field of view dynamically based on the
heterogeneous sensing (…wherein Baxter, in [0010], teaches a dynamically defined super-
pixel which selectively generates a control signal for elements of the dynamically defined
super-pixel. [0013] further teaches, with regards to fig. 1a, high-resolution pixels in a
"foveal" region as well as lower-resolution superpixels that are the result of
agglomeration of "standard" pixels.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a processor can selectively determine a
pixel with a high spatial resolution on regions of interest in a dynamic manner…).
59. Regarding claim 37, Saa in view of Pathak and Baxter teaches the method of claim 33 (see claim 33 above), wherein
the heterogeneous sensing is configured to sample more frequently from a region of
interest from said data to provide a sampling rate associated with said region, wherein said
region associated with a higher sampling rate can be dynamically movable and resizable on the
stable field of view, wherein different regions comprises different sampling rates (… Baxter, in
[0013] with regards to fig. 1a, teaches high-resolution pixels in a "foveal" region as well
as lower-resolution superpixels that are the result of agglomeration of "standard" pixels; [0027] in accordance with fig. 1b, teaches two high-spatial resolution regions (14, 15) that
may be dynamically changed; wherein, in accordance with fig. 1a, superpixel 10 defines
regions of the field of view that are of background peripheral regions which are not as
significant as image components which are sampled by high-spatial resolution foveal
pixels 11.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a variable acuity superpixel imager
(VASI) could have been implemented as a heterogeneous method of sensing received
light by an imager with different resolution, so to provide better definition to particular
areas of an imager, as such by defining certain regions with a higher-resolution than
other regions, as taught by Baxter…).
60. Claims 36 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Saa-
Garriga et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and Baxter et al. and (US 2004/0095492 A1; further referred to as Baxter) and further view of Calabretta (Mapping on the HEALPix grid).
61. Regarding claim 36, Saa in view of Pathak and Baxter teaches the method of claim 33 (see claim 33 above), wherein the heterogeneous sensing is applied in relation to a HEALPix projection or a double pixelisation (…wherein Calabretta teaches HEALPix projection or a double pixelisation, see Fig. 4 of Calabretta, it would be feasible to combine the concept to the teaching of Baxter wherein heterogeneous sensing can be implemented.
Therefore, it would have been obvious to one skilled in the art before the effective
filing date of the claimed invention that a variable acuity superpixel imager (VASI) could
have been implemented as a heterogeneous method of sensing received light by an imager with different resolution, so to provide better definition to particular areas of an
imager…).
62. Regarding claim 38, Saa in view of Pathak and Baxter teaches the method of claim 33 (see claim 33 above), further comprising:
dividing one or more HEALPix pixels of said data to increase spatial resolution of the
stable field of view (…Baxter, in [0013] with regards to fig. 1a, teaches high-resolution
pixels in a "foveal" region as well as lower-resolution superpixels that are the result of
agglomeration of "standard" pixels; further, Calabretta teaches HEALPix double-
pixelisation with increased interpolated pixels.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that increased resolution can be achieved by
means of double-pixelisation as taught by Calabretta, thus to provide better focus on
certain regions of focus…).
63. Claims 39-40 are rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga
et al. (US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and Shichijo et al. (US 2019/0318151 A1; further referred to as Shichijo).
64. Regarding claim 39, Saa in view of Pathak teaches the method of claim 1 (see claim 1 above), further comprising:
identifying an area of interest on an encoded stable field of view based on a N-1th data
frame of said data (…wherein Saa teaches the encoding of a stable field of view in
accordance with claim 1; Shichijo teaches an area determination method, as taught in
[0041]…),
wherein said data comprise at least a plurality of data frames (…wherein Shichijo , in
[0040-0041], teaches the storing of image data of image frames thus to be read and to
detect an area…);
mapping said area of interest to a Nth data frame of said data (…wherein [0041]
teaches area detector which uses a template matching method to search for a particular
area, “moving a position of a face reference template stepwise with respect to the image
data at a predetermined number of pixel intervals (thus viewed as mapping…); and
extracting a subset of data from said data based on the mapping (…wherein [0041]
further teaches that when an image area is matched to a reference template, the area is
extracted (a rectangular frame is used to extract the face image area.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that an area of interest (e.g. an area wherein
a particular object may be located) may be determined to be identified, mapped, and
extracted as taught by Shichijo so to effectively focus an area of interest…).
65. Regarding claim 40, Saa in view of Pathak and Shichijo teaches the method of claim 39 (see claim 39 above), wherein the extracted subset of data is represented by a 2D image independent of the encoded stable field of view (…wherein Shichijo, in [0097], teaches the positions of feature points as objects to be detected on a two-dimensional plane.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that an area of interest (e.g. an area wherein
a particular object may be located) may be determined to be identified, mapped, and
extracted as taught by Shichijo so to effectively focus an area of interest…).
66. Claim 41 is rejected under 35 U.S.C. 103 as being unpatentable over Saa-Garriga et al.
(US 2021/0142452 A1; further referred to as Saa) in view of Pathak, S. et al.; Spherical Video Stabilization by Estimating Rotation from Dense Optical Flow Fields (further referred to as Pathak) and Baxter et al. (US 2004/0095492 A1; further referred to as Baxter) and further view of Wissenbach et al. (US 2018/0332226 A1; further referred to as Wissenbach).
67. Regarding claim 41, the method of claim 39 (see claim 39 above), wherein the mapping is at least partially adapted to encode the stable field of view (…Saa, in [0051], teaches pixels forming frames of the 360-degree image may be indexed to a three-dimensional (3D) coordinate system defining locations of respective pixels on a surface of a virtual sphere…), wherein
the mapping is continuously updated to implement maximal-resolution heterogeneity (…wherein Saa, in [0051], teaches pixels forming frames of the 360-degree image may be indexed to a three-dimensional (3D) coordinate system defining locations of respective pixels on a surface of a virtual sphere; Baxter teaches an imaging system, as taught in [0009], including a VASI subsystem with the capability of retaining the highest possible spatial resolution on regions of interest that are important to the overall system; Wissenbach, additionally in [0047], teaches the capturing of live video by a group cameras in one or more consecutive frames.
Therefore, it would have been obvious to one of ordinary skill in the art before the
effective filing date of the claimed invention that a group of cameras can be coordinated
to provide a multidirectional view of an environment with the need to be observed in live
mode and thus be processed accordingly…).
Conclusion
68. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SURAFEL YILMAKASSAYE whose telephone number is (703)756-1910. The examiner can normally be reached Monday-Friday 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TWYLER HASKINS can be reached at (571)272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SURAFEL YILMAKASSAYE/Examiner, Art Unit 2639
/TWYLER L HASKINS/Supervisory Patent Examiner, Art Unit 2639
Prosecution Insights