Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 re rejected under 35 U.S.C. 103 as being unpatentable over Keegan (US 20240371138) in view of Zhang et al. (A Comprehensive Survey on Segment Anything Model for Vision and Beyond, arXiv:2305.08196v2v2 [cs.CV], 19 May 2023, pages 1-24, USPTO 892).
Regarding claims 1 and 14, Keegan discloses a method of automated defect recognition (Keegan Figs. 1 and 5A, and paragraphs 0005-0006, 0072 disclose pipe inspection tool and paragraph 0036 Keegan disclose In the example of FIG. 1, server 110 receives environment data from the inspection tool 105. In some embodiments, server 110 uses the received environmental data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe),
an article of manufacture configures to contain a set of instructions that are readable by a computer, the article of manufacture configured on a non-volatile memory, the set of instructions including a method written to perform (Keegan Figs, 1-2 and 5A and paragraphs 0021-0022 and 0072):
receiving a set of data related to piping formation (Keegan Fig. 1 and 5A, paragraphs 0046 and 0052, Keegan disclose, in some embodiments, the determined labels are association with project data, such as a jobsite or date and time of an inspection. For example, the determined label may be used to tag a jobsite as “this jobsite had two cracks identified and the pipe was PVC.” In some embodiments, the determined labels are associated with a particular region of a particular frame or frames. For instance, cracks are present on both sides of a pipe and paragraph 0052 Keegan disclose some embodiments, a variation of the sensor's 102 detection wavelength or through sonar imaging may seeing through allow non-metallic pipes (such as far IR imaging). This would assist in allowing detection of impending problems (such as a nearby root that is about to break the pipe) in addition to existing problems. In some embodiments, the inspection tool 105 broadcasts audio signals into a pipe to measure sounds, which is provided within the environment data for processing by the machine learning classifier 120. This obviously corresponds to receiving a set of data related to piping formation);
preparing a visual representation of the data (Keegan Fig. 1 and 5A, paragraphs 0046 and 0052, Keegan disclose, in some embodiments, the determined labels are association with project data, such as a jobsite or date and time of an inspection. For example, the determined label may be used to tag a jobsite as “this jobsite had two cracks identified and the pipe was PVC.” In some embodiments, the determined labels are associated with a particular region of a particular frame or frames. For instance, cracks are present on both sides of a pipe and paragraph 0052 Keegan discloses some embodiments, a variation of the sensor's 102 detection wavelength or through sonar imaging may see through allow non-metallic pipes (such as far IR imaging). This would assist in allowing detection of impending problems (such as a nearby root that is about to break the pipe) in addition to existing problems. In some embodiments, the inspection tool 105 broadcasts audio signals into a pipe to measure sounds, which is provided within the environment data for processing by the machine learning classifier 120. This obviously corresponds to preparing a visual representation of the data),
placing at least one positive data point on the visual representation (Keegan Fig. 1 5A and 6A-6B paragraph 0072 Keegan discloses pipe inspection tool, and paragraph 0036 Keegan discloses In the example of FIG. 1,tserver110 receives environmental data from the inspection tool 105. In some embodiments, server 110 uses the received environmental data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback and (re)training of the algorithm. In the system of Keegan it is obvious to place at least one positive data point on the visual representation because Keegan discloses include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) label may be used to label as a positive or negative such as one or more label not crack such as to place one or more label i.e. not crack);
placing at least two negative data points on the visual representation (Keegan Fig. 1, 5A and 6A-6B, paragraph 0072 Keegan discloses pipe inspection tool, and paragraph 0036 Keegan discloses In the example of FIG. 1, server 110 receives environmental data from the inspection tool 105. In some embodiments, server 110 uses the received environmental data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback and (re)training of the algorithm. In the system of Keegan it is obvious to place at least two negative data points on the visual representation such as use label to label two or more cracks);
feeding data of the at least one positive data point, the at least two negatives
data points and visual representation of the data into a Model (Keegan Figs. 1, 6A-6B and paragraph 0036 discloses FIGs.1, the server 110 receives environmental data from the inspection tool 105. In some embodiments, server 110 uses the received environmental data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback and (re)training of the algorithm, This obviously corresponds to feeding data of the at least one positive data point, the at least two negatives data points and visual representation of the data into a Model);
producing a mask with the Model, wherein the mask identifies defects in the data (Keegan, Figs. 1, 5A and 6A-6B, paragraph 0036 disclose, The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback and (re)training of the algorithm and Keegan paragraph 0044 discloses in some embodiments, the external device 107 may augment video in real-time via localization algorithms, such as YOLO, Mask R-CNN, R-CNN, Faster R-CNN, and so forth. Such an argument can be a box, mask, or outline. In some embodiments, by employing computer vision, measurements of various identified aspects and objects (e.g., cracks, defects, roots, and so forth) can be determined. All this obviously corresponds to producing a mask with the Model, wherein the mask identifies defects in the data);
As stated above, Keegan discloses artificial intelligence model (Keegan Fig. 1 and 6A-6B paragraph 0036 disclose In some embodiments, server 110 uses the received environmental data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe and paragraph 0044 disclose discloses in some embodiments, the external device 107 may augment video in real-time via localization algorithms, such as YOLO, Mask R-CNN, R-CNN, Faster R-CNN, and so forth. Such an argument can be a box, mask, or outline. In some embodiments, by employing computer vision, measurements of various identified aspects and objects (e.g., cracks, defects, roots, and so forth) can be determined i.e., Keegan is classifying/localizing and segmenting e.g., cracks, defects, roots, and so forth).
However Keegan has not explicitly disclosed large foundation model for classifying/localizing, i.e., segmentation.
In the same field of endeavor artificial intelligence model for learning/training and generating mask i.e., segmenting i.e., classifying and localizing task (Zhang, Figs. 1 and 6, page 1, Abstract and section 1. INTRUCTION. disclose artificial intelligence Foundation model segmenting anything (SAM) for generating mask i.e., segmenting anything Foundation model, page 2, section 2.2 Foundation Model right-column disclose large Foundation model and Zhang page 5, section 3.2 Real World Scenes right column, disclose large Foundation model and second paragraph disclose defect assessment of infrastructure and detect cracks which is illustrated in Fig. 6 shown and disclose unlabeled image and labeled image and Foundation model for generating mask).
Therefore, it would be obvious before the filing data of the claimed invention to use a large Foundation model as shown by Zhang and replace the model in the system of Keegan shown in paragraph 0044 of Keegan because such a system provides artificial intelligence model for segmenting anything and generating mask with wide variety of task as stated by Zhang in Abstract, section 1 INTRUCTION and section 5. CONCLUSION.
Regarding claims 2 and 15 Keegan discloses the data received is ultrasonic data (Keegan paragraph 0052 sonar imaging which obviously include ultrasonic sensor and ultrasonic sensor is well known in the art).
Regarding claims 3 and 16 Keegan discloses the data is a previously completed analysis log that has been previously annotated. (Keegan in paragraph 0036 disclose data and label data for training/ learning and Zhang Fig. 6 disclose image data and labeled image for learning/training. In the system of Keegan and Zhang it would be obvious to store labeled data and analysis data to log so the data can be used later for learning and training)
Regarding claims 4 and 17 Keegan discloses the data in a digital format (Keegan Figs 1-2 and 6A-6B and paragraphs 0021-0022, 0072 and 0036 it is obvious in the system of Keegan the data is in a digital format).
Regarding claims 5 and 18 Keegan discloses the data stored on one non- volatile device, a computer server, a personal computer and a network (Keegan Figs 1-2 and 6A-6B and paragraphs 0021-0022, 0072 and 0036, the data stored on one non- volatile device, a computer server and a network).
Regarding claim 6 Keegan discloses the visual representation is on a computer monitor (Keegan Figs, 1 and 6A-6B paragraph 0036 in the system of Keegan it is obvious that the visual representation is on a computer monitor).
Regarding claim 7 Keegan discloses the placing of at- least one positive data point or the placing of the at least two negative data points is through an annotation tool (Keegan Figs. 1 and 6A-6B, paragraph 0036 In the example of FIG. 1, the server 110 receives environment data from the inspection tool 105. In some embodiments, the server 110 uses the received environment data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, the server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environments and determine labels for these aspects based on the input environment data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback. This obviously corresponds to the placing of at- least one positive data point or the placing of the at least two negative data points is through an annotation too ).
Regarding claim 8 Keegan discloses the placing of at least one positive data point or the placing of the at least two negative data points is through an annotation tool and an artificial intelligence computer program (Keegan Figs. 1 and 6A-6B, paragraph 0036 In the example of FIG. 1, the server 110 receives environment data from the inspection tool 105. In some embodiments, server 110 uses the received environment data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environments and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback. This obviously corresponds to the placing of at least one positive data point or the placing of the at least two negative data points is through an annotation tool and artificial intelligence computer program)
Regarding claim 9 Keegan discloses the placing of the at least one positive data point or the placing of the at least two negative data points is through an annotation tool (Keegan Figs. 1 and 6A-6B, paragraph 0036 In the example of FIG. 1, the server 110 receives environment data from the inspection tool 105. In some embodiments, server 110 uses the received environment data as additional training examples (e.g., when the actual value or classification is also known). In other embodiments, the server 110 sends the received environment data to the trained machine learning classifier 120. The machine learning classifier 120 is then employed to classify various aspects of the environments and determine labels for these aspects based on the input environment data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback. This obviously corresponds to the placing of the at least one positive data point or the placing of the at least two negative data points is through an annotation tool).
Regarding claim 10 Keegan discloses the defects in the data are related to one of corrosion on the piping or mechanical damage of the piping (Keegan paragraph 0036 and 0044 disclose and paragraph 0044 disclose discloses in some embodiments, the external device 107 may augment video in real-time via localization algorithms, such as YOLO, Mask R-CNN, R-CNN, Faster R-CNN, and so forth. Such an argument can be a box, mask, or outline. In some embodiments, by employing computer vision, measurements of various identified aspects and objects (e.g., cracks, defects, roots, and so forth) can be determined i.e., Keegan is classifying/localizing and segmenting e.g., cracks, defects, roots, and so forth obviously include and corresponds to discloses the defects in the data are related to one of corrosion on the piping or mechanical damage of the piping)
Regarding claim 11, Keegan discloses deleting at least one of the positive points and the negative points (Keegan Figs. 1 and 6A-6B, paragraph 0036 disclose The machine learning classifier 120 is then employed to classify various aspects of the environment and determine labels for these aspects based on the input environmental data. For example, the trained machine learning classifier 120 may process the received image data to determine that a crack has formed in a pipe. The server 110 may then transmit a suggested label for the determined classification (e.g., “Crack”) to the external device 107. The external device 107 may prompt the user if they would like to include the label. The external device 107 may also prompt the user with additional options such as, for example, “Not a Crack,” “Something Else,” “Ignore,” “Unsure”, and so forth. Additionally, a user's selection of certain options (e.g., “Unsure) may trigger a prompt to send or sending the image or footage to another user or a network of users. Moreover, selecting an option (e.g., “Crack” or “Not a Crack”) may be used as a positive or negative example for feedback and (re)training of the algorithm. In the system of Keegan the user is interactively labeling and it would be obvious in the system of Keegan it would be obvious to delete at least one of the positive points and the negative points).
Regarding claim 12 Keegan discloses saving the mask generated by the Large Foundation Model. (Zhang disclose in section 1 INTRODUCTION large Foundation model for artificial intelligence for training using large scale web-scaled data and note: Fig. 6: generated mask and the system of Zhang is obviously computer and net based system as disclose in section 1. Introduction, therefore it would be obvious to saving the mask generated by the Large Foundation Model in the system memory)
Regarding claim 13 Keegan discloses he saving mask generated by the Large Foundation Model is to a memory arrangement. (Zhang disclose in section 1 INTRODUCTION large Foundation model for artificial intelligence for training using large scale web-scale data and note: Fig. 6: generated mask and the system of Zhang is obviously computer and net based system as disclose in section 1. Introduction, therefore, it would be obvious to save the mask generated by the Large Foundation Model in the system memory arrangement)
Communication Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHRAT I SHERALI whose telephone number is (571)272-7398. The examiner can normally be reached Monday-Friday 8:00AM -5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached on 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
ISHRAT I. SHERALI
Examiner
Art Unit 2667
/ISHRAT I SHERALI/Primary Examiner, Art Unit 2667
Prosecution Insights