Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed on 5 November, 2024.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 5 November, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 15 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 15 contains the trademark/trade names “PCIe” and “Bluetooth” on lines 3 and 5 respectively. Where a trademark or trade name is used in a claim as a limitation to identify or describe a particular material or product, the claim does not comply with the requirements of 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph. See Ex parte Simpson, 218 USPQ 1020 (Bd. App. 1982). The claim scope is uncertain since the trademark or trade name cannot be used properly to identify any particular material or product. A trademark or trade name is used to identify a source of goods, and not the goods themselves. Thus, a trademark or trade name does not identify or describe the goods associated with the trademark or trade name. In the present case, the trademark/trade name is used to identify/describe a wireless communication protocol (Bluetooth) and a type of connection for physical data transmission (PCIe) and, accordingly, the identification/description is indefinite.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of pre-AIA 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a) the invention was known or used by others in this country, or patented or described in a printed publication in this or a foreign country, before the invention thereof by the applicant for a patent.
(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country, more than one year prior to the date of application for patent in the United States.
Claims 1 – 8, and 11 – 19 are rejected under pre-AIA 35 U.S.C. 102(a)(2) as being anticipated by Chen et al (U.S. Patent Publication No. 2019/0130580 A1, hereinafter “Chen”).
Regarding claim 1, Chen teaches an image processing method comprising:
receiving a first image of an image stream having a stream frame rate (¶ 0136: The video analytics system 100 receives video frames 102 from a video source 130… The video frames 102 can be part of one or more video sequences.; ¶ 0159: For illustrative purposes, it is assumed that the video analytics system 1200 obtained the first frame of a video sequence at time TM…);
passing the received first image to image processing functionality (Figure 2, Ref. No. 202A, 204A; ¶ 0143: FIG. 2 is an example of the video analytics system (e.g., video analytics system 100) processing video frames across time t. As shown in FIG. 2, a video frame A 202A is received by a blob detection system 204A. The blob detection system 204A generates foreground blobs 208A for the current frame A 202A. After blob detection is performed, the foreground blobs 208A can be used for temporal tracking by the object tracking system 206A.; ¶ 0256: To make the whole system run smoothly, assuming the system frame rate is fr, it is proposed to apply the high complexity detector to key input frames.; ¶ 0259: For illustrative purposes, it is assumed that the video analytics system 1200 obtained the first frame of a video sequence at time TM, and that the system 1200 can start the high complexity detector 1708 (using the deep learning system 1208) from frame M immediately.);
receiving a plurality of subsequent images of the image stream (¶ 0144: When a next video frame N 202N is received, the blob detection system 204N generates foreground blobs 208N for the frame N 202N.; ¶ 0281: In some examples, the process 1800 includes obtaining a subsequent video frame that is obtained later in time than the video frame.);
storing the received plurality of subsequent images in a fast-track image buffer (¶ 0257: Assuming the motion-based video analytics processes (e.g., those described with respect to FIG. 1-FIG. 4) can already be processed in real-time in order to sync-up between the high complexity deep learning based detector results and the motion-based video analytics results, the system 1200 can buffer N+1 frames in memory.);
subsequent to receiving at least one subsequent image, receiving from the image processing functionality an indication of one or more features within the first image (Figure 17; ¶ 0259: However, the motion-based video analytics components (blob detection and tracking, denoted as the VA Engine 1704 in FIG. 17) have to wait for the deep learning detection results and is thus delayed… In the example shown in FIG. 17, the number of frames N is equal to 8. With N=8, no frames are fed into the VA Engine 1704 until time TM+8. At time TM+8, since detection results from frame M are ready, the detector bounding boxes (BBDetector) are fed into the VA engine 1704 together with frame M.); and
tracking the one or more features across the plurality of subsequent images stored within the fast-tracking image buffer using a tracking process configured to process each of the subsequent images at a processing frame rate higher than the stream frame rate (¶ 0007: Once a tracker has at least one bounding region that is a high confidence bounding region detected in a key frame, in the subsequent frames, the tracker is referred to herein as a high confidence tracker.; ¶ 0216: once a tracker has at least one bounding box that is a high confidence bounding box detected in a key frame, the tracker is denoted as a high confidence tracker in the subsequent frames. In such cases, to continue tracking blobs (and associated objects) using a high confidence tracker across video frames, the object tracking system 1206 can perform data association between the bounding box of the high confidence tracker in the previous frame and a bounding box of a detected blob in a current frame (detected using the blob detection system 1204).).
Regarding claim 2, Chen teaches the method of claim 1.
Additionally, Chen teaches wherein the image processing functionality identifies the one or more features in the received first image (¶ 0259: For illustrative purposes, it is assumed that the video analytics system 1200 obtained the first frame of a video sequence at time TM, and that the system 1200 can start the high complexity detector 1708 (using the deep learning system 1208) from frame M immediately… wait for the deep learning detection results and is thus delayed.).
Regarding claim 3, Chen teaches the method of claim 1.
Additionally, Chen teaches wherein the image processing functionality identifies the one or more features in the received first image using a machine learning process (¶ 0210: The deep learning system 1208 can implement a complex object detector. For example, the complex object detector can be implemented using one or more trained neural networks (e.g., a deep learning network) to one or more of the frames 1202 of the received video sequence to locate and classify objects in the one or more frames.).
Regarding claim 4, Chen teaches the method of claim 1.
Additionally, Chen teaches wherein passing the received first image to the image processing functionality comprises:
passing the first image to the processing functionality implemented at a remote computing device over a communication interface (¶ 0004: The system with the video analytics can be on a camera device and/or on a server.; ¶ 0137: In some embodiments, the video analytics system 100 and the video source 130 can be part of the same computing device. In some embodiments, the video analytics system 100 and the video source 130 can be part of separate computing devices.; ¶ 0295: In some examples, a camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives the captured video data. The computing device may further include a network interface configured to communicate the video data.).
Regarding claim 5, Chen teaches the method of claim 1.
Additionally, Chen teaches further comprising:
after tracking the one or more features across all of the subsequent images stored in the fast-tracking buffer, tracking the one or more features across newly received images of the image stream (Figure 17; ¶ 0259: However, the motion-based video analytics components (blob detection and tracking, denoted as the VA Engine 1704 in FIG. 17) have to wait for the deep learning detection results and is thus delayed… In the example shown in FIG. 17, the number of frames N is equal to 8. With N=8, no frames are fed into the VA Engine 1704 until time TM+8. At time TM+8, since detection results from frame M are ready, the detector bounding boxes (BBDetector) are fed into the VA engine 1704 together with frame M. At time TM+9 frame M+9 is fed into the high complexity detector and meanwhile frame M+1 is fed into the VA engine 1704 (without corresponding detector results from the detector 1708 for frame M+9).; Examiner’s note: Figure 17 time stamp TM+9 where frames M+1 is being entered into the VA Engine 1704 after the Detector 1708 has provided the detection results. The Detector 1708 is then utilized to begin determining detection results on frame M+9. The VA engine then tracks frames M+1 – M+8 at which time the detection results from frame M+9 will be provided to the VA Engine 1704 and tracking of subsequent frames will be performed.).
Regarding claim 6, Chen teaches the method of claim 5.
Additionally, Chen teaches wherein tracking the one or more features across the newly received images of the image stream uses the tracking process used for tracking the one or more features across the subsequent images stored within the fast-tracking image buffer (¶ 0262: In some cases, if the detector 1704 finishes slower than expected (e.g., when results of frame M+X are ready), in which case the VA engine 1704 would already be dealing with frame M+X+D (wherein M, X, and D are all positive integers), backward tracking similar to that described above for non-key frames may apply.).
Regarding claim 7, Chen teaches the method of claim 5.
Additionally, Chen teaches wherein tracking the one or more features across the newly received images of the image stream uses a different tracking process than used for tracking the one or more features across the subsequent images stored within the fast-tracking image buffer (¶ 0262: In some cases, if the detector 1704 finishes slower than expected (e.g., when results of frame M+X are ready), in which case the VA engine 1704 would already be dealing with frame M+X+D (wherein M, X, and D are all positive integers), backward tracking similar to that described above for non-key frames may apply.), wherein the tracking process for tracking the one or more features across the newly received images of the image stream has a slower processing frame rate than the processing frame rate of the tracking process used for tracking the one or more features across the subsequent images stored within the fast-tracking image buffer (¶ 0263: In some cases, if a detector finishes slower than expected (e.g., when results of frame M+X are ready), the video analytics system 1200 can monitor the delay and can adjust the input parameter N to achieve better synchronization.).
Regarding claim 8, Chen teaches the method of claim 1.
Additionally, Chen teaches further comprising:
receiving new images of the image stream while tracking the one or more features across the subsequent images stored within the fast-tracking image buffer (Figure 17; ¶ 0259: In the example shown in FIG. 17, the number of frames N is equal to 8. With N=8, no frames are fed into the VA Engine 1704 until time TM+8. At time TM+8, since detection results from frame M are ready, the detector bounding boxes (BBDetector) are fed into the VA engine 1704 together with frame M. At time TM+9 frame M+9 is fed into the high complexity detector and meanwhile frame M+1 is fed into the VA engine 1704 (without corresponding detector results from the detector 1708 for frame M+9).); and
storing the new images in the fast-tracking image buffer (Figure 17, TM+9; ¶ 0259: In the example shown in FIG. 17, the number of frames N is equal to 8. With N=8, no frames are fed into the VA Engine 1704 until time TM+8. At time TM+8, since detection results from frame M are ready, the detector bounding boxes (BBDetector) are fed into the VA engine 1704 together with frame M. At time TM+9 frame M+9 is fed into the high complexity detector and meanwhile frame M+1 is fed into the VA engine 1704 (without corresponding detector results from the detector 1708 for frame M+9).).
The rejection of method claim 1 above applies mutatis mutandis to the corresponding limitations of device claim 11 while noting that the rejection above cites to both method and device disclosures.
For the device limitations of claim 11 see Chen’s teaching on:
An image processing device comprising:
A processor capable of executing instructions (¶ 0015: In another example, a non-transitory computer
Readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processor to…); and
A memory storing instructions which when executed by the processor configure the image processing device to provide a method comprising (¶ 0015: In another example, a non-transitory computer
Readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processor to…)…
The rejection of method claim 2 above applies mutatis mutandis to the corresponding limitations of device claim 12 while noting that the rejection above cites to both method and device disclosures.
The rejection of method claim 3 above applies mutatis mutandis to the corresponding limitations of device claim 13 while noting that the rejection above cites to both method and device disclosures.
The rejection of method claim 4 above applies mutatis mutandis to the corresponding limitations of device claim 14 while noting that the rejection above cites to both method and device disclosures.
Regarding claim 15, Chen teaches the method of claim 14.
Additionally, Chen teaches wherein the communication interface is at least one of:
A PCIe communication interface;
A USB interface;
A Bluetooth interface;
A wired network interface (¶ 0540: Destination device may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server.); and
A wireless network interface (¶ 0137: In some examples, the computing device (or devices) can include one or more wireless transceivers for wireless communications.).
The rejection of method claim 5 above applies mutatis mutandis to the corresponding limitations of device claim 16 while noting that the rejection above cites to both method and device disclosures.
The rejection of method claim 6 above applies mutatis mutandis to the corresponding limitations of device claim 17 while noting that the rejection above cites to both method and device disclosures.
The rejection of method claim 7 above applies mutatis mutandis to the corresponding limitations of device claim 18 while noting that the rejection above cites to both method and device disclosures.
The rejection of method claim 8 above applies mutatis mutandis to the corresponding limitations of device claim 19 while noting that the rejection above cites to both method and device disclosures.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al (U.S. Patent Publication No. 2019/0130580 A1, hereinafter “Chen”) in view of Renschler et al (U.S. Patent Publication No. 2024/0289916 A1, hereinafter “Renschler”).
Regarding claim 9, Chen teaches the method of claim 1.
Chen does not explicitly teach further comprising one or more of: removing subsequent images stored in the fast-tracking image buffer once processed by the tracking process; and marking subsequent images stored in the fast-tracking image buffer as safe for removal once processed by the tracking process.
However, Renschler does teach further comprising one or more of:
removing subsequent images stored in the fast-tracking image buffer once processed by the tracking process (¶ 0066: Each image frame buffer can temporarily store a single image frame that is captured by the image sensor 130 and/or processed by the image processor 150. In some examples, an image frame buffer can be, or can include, a circular buffer, a circular queue, a cyclic buffer, a ring buffer, or a combination thereof. In some examples, the one or more image buffers 170 can receive and store partial image frame data from an image frame before the entirety of the image has completed capturing. In some examples, image frame data from a new image frame overwrites older existing image buffer data in an image frame buffer.); and
marking subsequent images stored in the fast-tracking image buffer as safe for removal once processed by the tracking process.
Renschler is considered to be analogous art as it pertains to image processing utilizing frame buffers to store data. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for applying complex object detection in a video analytics system (as taught by Chen) and the low latency frame delivery device (as taught by Renschler) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Renschler utilizes partial frame delivery (shown in figure 3 and 5) to improve the latency and consistency of the system significantly (See ¶ 0093).
This motivation for the combination of Chen and Renschler is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
The rejection of method claim 9 above applies mutatis mutandis to the corresponding limitations of device claim 20 while noting that the rejection above cites to both method and device disclosures.
Claims 10 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al (U.S. Patent Publication No. 2019/0130580 A1, hereinafter “Chen”) in view of Becker et al (D. E. Becker, A. Can, J. N. Turner, H. L. Tanenbaum and B. Roysam, "Image processing algorithms for retinal montage synthesis, mapping, and real-time location determination," in IEEE Transactions on Biomedical Engineering, vol. 45, no. 1, pp. 105-118, Jan. 1998, doi: 10.1109/10.650362, hereinafter “Becker”)
Regarding claim 10, Chen teaches the method of claim 1.
Chen does not explicitly teach wherein image stream is of a patient's eye, the method further comprising: using the one or more identified features tracked across images of the image stream in treatment of an eye condition.
However, Becker does teach wherein image stream is of a patient's eye (Figure 1; Abstract: The algorithm for locating vasculature landmarks performed robustly at a speed of 16–30 video image frames/s depending upon the field on a Silicon Graphics workstation.; Page 105, Col. 2, ¶ 3: Automated synthesis of wide-area retinal montage and map: This algorithm is used to combine a number of fundus camera images of a patient’s retina into a montage with a consistent coordinate system.), the method further comprising:
using the one or more identified features tracked across images of the image stream in treatment of an eye condition (Page 105, Col. 2, ¶ 1: The key to effective and lasting treatment is the identification of the full extent of the CNV, complete cauterization of the CNV by accurately aiming an appropriate amount of optical energy while ensuring that healthy tissue is not cauterized.; Page 105, Col. 2, ¶ 2: Among other functions, this instrument is intended to perform montaging, mapping, real-time tracking of the retina, and spatial dosimetry of the applied laser energy.; Page 105, Col. 2, ¶ 4: This algorithm is designed to be used in a computer-assisted laser delivery system to determine the location of a live retinal fundus video image relative to the wide-area retinal montage map, in real time. In other words, this algorithm is used to track the patient’s retina relative to the retinal map, and provide control signals to a computer-controlled laser delivery system.).
Becker is considered to be analogous art as it pertains to image processing of a live video. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for applying complex object detection in a video analytics system (as taught by Chen) and the image processing algorithms for retinal montage synthesis (as taught by Becker) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Becker uses an improved point-matching algorithm which results in an overall speed improvement of approximately 180-200 times. (See page 110, Col. 2, ¶ 2).
This motivation for the combination of Chen and Becker is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
The rejection of method claim 10 above applies mutatis mutandis to the corresponding limitations of device claim 21 while noting that the rejection above cites to both method and device disclosures.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW B. JONES/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights