Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-3, 5-9, 11 and 14-18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 6 is objected to because of the following informalities: “that the first and second feature maps” in lines 2-3 should read “that the at least one of the first and the second feature maps”” in line 2 should read “the artificial neural network” Appropriate correction is required.
Claim 11 is objected to because of the following informalities: “an artificial neural network” in line 2 should read “the artificial neural network” Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 16 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 16 recites at line 2 “an encoder”. Claim 16 depends from claim 14. Claim 14 recites at line 17 “an encoder”. Claim 16 further recites at lines 3-4 “the encoder”. It is unclear as to which of the above recitals of “an encoder” in claim 14 lines 17 and claim 16 line 2, the recital of “the encoder” at lines 3-4 in claim 16 refers to?. Amendments/clarification are required. Claim 17-18 depends directly or indirectly from claim 16, therefore they are rejected.
Claim 3 recites the limitation "the polygon" in line 2. There is insufficient antecedent basis for this limitation in the claim. Claim 6 depends from claim 3, therefore it is rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 9, 14 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al., (US20220261593) hereafter Yu in view of NPL12 (SENSE: a Shared Encoder Network for Scene-flow Estimation, Huaizu Jiang et al., IEEE, 2019, Pages 3194-3203) hereafter NPL12.
1. Regarding claim 1, Yu discloses a method (figs 1-4, 7, 11A-11C, 14 and 20, para 0273 shows and discloses a method) for fusing sensor data, comprising the following steps:
a) receiving input sensor data, wherein the input sensor data (figs 1-2, 4, 7, 11A-11C and 20 shows and discloses receiving input images 102, 104 from the sensors (fig 11A shows the sensors such as cameras, radar, Lidar etc.) meeting the above claims limitations, examiner notes that the specifics of input sensor data are not required by the current claim) comprise:
a first representation which comprises a first region of a scene
PNG
media_image1.png
191
320
media_image1.png
Greyscale
, and
a second representation which comprises a second region of the scene
PNG
media_image2.png
180
326
media_image2.png
Greyscale
, wherein the first and second regions overlap one another, but are not identical (fig 1 shows the trucks as the first region in the scene in the second image and the second region with a truck in the scene and the trucks in the second image (i.e the first region) is common (i.e overlap one another) in the first image (i.e the second region) of the scene meeting the claim limitations), and the first region is an overview region of the scene (fig 1 shows the trucks in the second image which is an overview region of the scene) and the second region is a partial region of the overview region of the scene (i.e the second region shows the partial region (i.e only containing a big truck) of the overview region (i.e the first region containing the overview of the scene) as seen in fig 1 meeting the claim limitations)
PNG
media_image3.png
499
740
media_image3.png
Greyscale
b) determining a first feature map with a first height and width on the basis of the first representation and determining a second feature map with a second height and width on the basis of the second representation (figs 2, 4 and paras 0073-0078, 0089, 0101 shows and discloses features maps 406 and 408 determined from the images 402/404 (202) respectively through feature pyramids 204B and 206B, examiner notes that the pyramids have the height and width meeting the above claim limitations, para 0074 discloses “In at least one embodiment, for a framework for object detection, instance segmentation, and semantic correspondence, an input image is denoted by I (e.g., input images 202), deep features are denoted by F (e.g., features determined by a feature pyramid 206B and a feature pyramid 204B” meeting the above claim limitations);
c) computing a first output feature map by a first convolution of the first feature map, and computing a second output feature map by a second convolution of the second feature map (figs 2, 4 and paras 0073-0078, 0089, 0101 shows and discloses computing a first output feature map by a first convolution of the first feature map, and computing a second output feature map by a second convolution of the second feature map, paras 0074 discloses “In at least one embodiment, for a framework for object detection, instance segmentation, and semantic correspondence, an input image is denoted by I (e.g., input images 202), deep features are denoted by F (e.g., features determined by a feature pyramid 206B and a feature pyramid 204B” and para 0076 discloses one or more convolution neural networks 206A and 204A extracting the features meeting the above claim limitations). Yu discloses video encoders and decoders in para 0291. Yu however is silent and fails to disclose d) computing, by an encoder of an artificial neural network, a fused feature map through element-by-element addition of the first and second output feature maps, wherein a position of the first and the second region with respect to one another is used to compute the fused feature map, such that elements in a region of overlap are added; e) determining, by the encoder of the artificial neural network, Advanced Driver Assistance System/Autonomous Driving (ADAS/AD)-relevant information using the fused feature map, wherein the artificial neural network includes multiple decoders for different ADAS/AD detection functions; and e) outputting the fused feature map.
NPL12 discloses d) computing, by an encoder of an artificial neural network, a fused feature map through element-by-element addition of the first and second output feature maps, wherein a position of the first and the second region with respect to one another is used to compute the fused feature map, such that elements in a region of overlap are added (figs 1-fig 2, pages 3196-3197 and 3199 col 1 shows and discloses d) computing, by an encoder of an artificial neural network, a fused feature map through element-by-element addition of the first and second output feature maps, wherein a position of the first and the second region with respect to one another is used to compute the fused feature map, such that elements in a region of overlap are added); e) determining, by the encoder of the artificial neural network, Advanced Driver Assistance System/Autonomous Driving (ADAS/AD)-relevant information using the fused feature map, wherein the artificial neural network includes multiple decoders for different ADAS/AD detection functions (pages 3196-3197, fig 2 shows artificial neural network with a single encoder (i.e shared encoder) for different decoding tasks related to ADAS/AD functions i.e decoder for disparity estimation and decoder for segmentation and decoder for occlusion estimation meeting the above claim limitations, examiner notes that the specifics of the different tasks are not required by the current claim); and e) outputting the fused feature map (fig 2 shows the output of the fused feature map (i.e the addition of the images/features maps and results outputted) meeting the claim limitations). Before the effective filing date of the invention was made, Yu and NPL12 are combinable because they are from the same filed of endeavor and are analogous art of image processing. The suggestion/motivation would be a compact and efficient model/method on pages 3196 col 2, and 3197 col2. Therefore, it would be obvious and within one of ordinary skill in the art to have recognized the advantages of NPL12 in the method/system of Yu to obtain the invention as specified in claim 1.
2. Regarding claim 2, Yu and NPL12 disclose the method according to Claim 1. Yu shows wherein the first and second output feature maps have the same height and width in the region of overlap (figs 2, 4 and paras 0073-0078, 0089, 0101 shows and discloses features maps 406 and 408 determined from the images 402/404 (202) respectively through feature pyramids 204B and 206B, examiner notes that the pyramids have the same height and width meeting the above claim limitations, para 0074 discloses “In at least one embodiment, for a framework for object detection, instance segmentation, and semantic correspondence, an input image is denoted by I (e.g., input images 202), deep features are denoted by F (e.g., features determined by a feature pyramid 206B and a feature pyramid 204B” meeting the above claim limitations).
3. Regarding claim 3, as best understood by the examiner, Yu and NPL12 disclose the method according to claim 1. Yu discloses and shows wherein height and width of the fused feature map are determined by the polygon which surrounds the first and the second feature map, and wherein the polygon is a rectangle (fig 2 para 0076 shows and
PNG
media_image4.png
203
450
media_image4.png
Greyscale
discloses wherein height and width of the fused feature map are determined by the polygon which surrounds the first and the second feature map, and wherein the polygon is a rectangle meeting the claim limitations, examiner notes that the polygon is a rectangle and it has height and width).
4. Regarding claim 5, Yu and NPL12 discloses the method according to Claim 1. Yu discloses further wherein the first representation has a first resolution and the second representation has a second resolution, wherein the second resolution is higher than the first resolution (figs 1-2, 4 and para 0076 discloses the combining of the low resolution, semantically strong features with high resolution, semantically weak features meeting the above claim limitations).
5. Regarding claim 9, Yu and NPL12 disclose the method according to Claim 1. Yu disclose further wherein the first and second feature maps each have a depth which depends on a resolution of at least one of the first representation or the second representation (paras 0076, 0166 discloses an embodiment wherein the camera may be a depth camera and the images with the resolution therefore input and processed would have a depth and would therefore disclose wherein the first and second feature maps each have a depth which depends on a resolution of at least one of the first representation or the second representation).
6. Claim 14 is a corresponding system claim of claim 1. See the corresponding explanation of claim 1. Yu shows and discloses system for fusing sensor data, comprising an input interface, a data processing unit and an output interface (figs 1-4, 7, 11A-11C, 14 and 20, paras 0158, 0206-0211 and 0273 shows and discloses a system comprising an input interface, a data processing unit and an output interface for performing the functions as recited in claim 14.
7. Regarding claim 16, as best understood by the examiner, Yu and NPL12 disclose the system according to Claim 14. NPL12 shows and discloses wherein the system comprises a convolutional neural network having an encoder and wherein the input interface, the data processing unit and the output interface are implemented in the encoder such that the encoder is configured to generate the fused feature map (figs 1-2, pages 3196-3197 encoder architecture with input interface (i.e input images are inputted), data processed by the CNN having a single encoder and multiple decoders (i.e data processing unit) and output images as seen in fig 2, (i.e 3201 shows and discloses computation, running time, model size and memory meeting the claim limitations of claim 16).
8. Regarding claim 17, Yu and NPL12 disclose the system according to Claim 16. NPL12 discloses further wherein the convolutional neural network comprises multiple decoders which are configured to realize different ADAS/AD detection functions at least on the basis of the fused feature map (pages 3196, 3197 shows and discloses wherein the convolutional neural network comprises multiple decoders which are configured to realize different ADAS/AD detection functions at least on the basis of the fused feature map i.e from the input images, see the arrows in the fig
.
PNG
media_image5.png
413
875
media_image5.png
Greyscale
meeting the claim limitations).
9. Regarding claim 18, Yu and NPL12 disclose the system according to Claim 17. NPL12 disclose the further comprising an ADAS/AD controller, wherein the ADAS/AD controller is configured to realize ADAS/AD functions at least on the basis of results of the ADAS/AD detection functions (pages 3196-3197, fig 2 shows artificial neural network with a single encoder (i.e shared encoder) for different decoding tasks related to ADAS/AD functions i.e decoder for disparity estimation and decoder for segmentation and decoder for occlusion estimation and figs 1-2 and pages 3194, 3197 shows and discloses the results of the decoders (output images) for ADAS/AD functions i.e disparity, segmentation and occlusion in autonomous driving meeting the above claim limitations).
Claims 11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Yu, NPL12 and in further view of NPL2 (Fused-Layer CNN Accelerators, Manoj Alwani et al., IEEE, 2016, Pages 1-12) hereafter NPL2.
10. Regarding claim 11, Yu and NPL12 discloses the method according to Claim 1. Yu and NPL12 both disclose and shows the method implemented by the convolution neural network (figs 1-4, 7, 11A-11C, 14 and 20) and figs 1-3 respectively. Yu and NPL12 however are silent and fails to disclose wherein the method is implemented in a hardware accelerator for an artificial neural network.
NPL2 discloses wherein the method is implemented in a hardware accelerator for an artificial neural network (Page 2 cols 1-2 discloses the method is implemented in a hardware accelerator for an artificial neural network). Before the effective filing date of the invention was made, Yu, NPL12 and NPL2 are combinable because they are from the same filed of endeavor and are analogous art of image processing. The suggestion/motivation would be an increased throughput, reduced data and maximum performance and highly efficient system/method on page 6 cols 1-2. Therefore, it would be obvious and within one of ordinary skill in the art to have recognized the advantages of NPL2 in the method of Yu and NPL12 to obtain the invention as specified in claim 11.
11. Regarding claim 15, Yu and NPL12 discloses the system according to Claim 14. Yu and NPL1 disclose and shows the method implemented by the convolution neural network/hardware (figs 1-4, 7, 11A-11C, 14 and 20) and figs 1-3 respectively). Yu and NPL12 however are silent and fails to disclose wherein the system comprise a CNN hardware accelerator, wherein the input interface, the data processing unit and the output interface are implemented in the CNN hardware accelerator.
NPL2 discloses wherein the system comprises a CNN hardware accelerator, wherein the input interface, the data processing unit and the output interface are implemented in the CNN hardware accelerator (Fig 3, Pages 2-3, page 5 cols 1-2 discloses the method is implemented in a hardware accelerator the for an artificial neural network with the input interface (initial computation layer), data processing unit (intermediate processing layer) and the output interface (i.e output features) meeting the above claim limitations). Before the effective filing date of the invention was made, Yu, NPL12 and NPL2 are combinable because they are from the same filed of endeavor and are analogous art of image processing. The suggestion/motivation would be an increased throughput, reduced data and maximum performance and highly efficient system/method on page 6 cols 1-2. Therefore, it would be obvious and within one of ordinary skill in the art to have recognized the advantages of NPL2 in the method of Yu and NPL1 to obtain the invention as specified in claim 15.
Examiner's Note: Examiner has cited figures, and paragraphs in the references as applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested for the applicant, in preparing the responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Examiner has also cited references in PTO892 but not relied on, which are relevant and pertinent to the applicant’s disclosure, and may also be reading (anticipatory/obvious) on the claims and claimed limitations. Applicant is advised to consider the references in preparing the response/amendments in-order to expedite the prosecution.
Allowable Subject Matter
Claims 7-8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAYESH PATEL whose telephone number is (571)270-1227. The examiner can normally be reached IFW Mon-FRI.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAYESH PATEL/
Primary Examiner
Art Unit 2677
/JAYESH A PATEL/Primary Examiner, Art Unit 2677
Prosecution Insights