DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/7/2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 12, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980).
Regarding claim 1, Zia teaches information processing method performed by a computer, the method comprising: attaching a first information tag to a target segment of video data on a basis of the video data obtained by capturing a video of a worker using an evaluation model according to a machine learning technique, the first information tag relating to a factory work (see figure 3, figure 4, figure 7, figure 8, Zia discusses annotating tasks captured in video of a worker using a trained machine learning model); and
wherein the first display image includes a comparison display including a first display area corresponding to the first information tag and a second display area corresponding to the second infom1ation tag, the first display area and the second display area extending along a common time axis corresponding to the time line of the video data, the first display area including an indication of a segment of the first information tag at a position corresponding to a time segment on the common time axis, and the second display area including an indication of a segment of the second information tag at a position corresponding to a time segment on the common time axis (see figure 4, figure 5, Zia discusses a user interface display that contains annotated tasks that are captured in videos of workers, the tasks are displayed in different time segments of a common timeline).
Huval teaches generating a first display image that shows both a second information tag and the first information tag simultaneously on the same time line of the video data, the second information tag relating to the factory work and being attached to a segment of the video data by a user manually without using the evaluation model, the first information tag being attached to the target segment using the evaluation model according to the machine learning technique (see figure 1, figure 3, figure 4, para. 0008, Huval discusses manual label annotation and automatic label annotation using a neural network at the same time T1).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia with Huval to derive at the invention of claim 1. The result would have been expected, routine, and predictable in order to perform work image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia in this manner in order to improve image annotation of a workplace by allowing a user to annotate images from a same time period. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia, while the teaching of Huval continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images within a time period and annotating the objects to analyze work progress. The Zia and Huval systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 12 is rejected as applied to claim 1 as pertaining to a corresponding device.
Claim 13 is rejected as applied to claim 1 as pertaining to a corresponding non-transitory computer readable medium.
Claims 2-6 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980) in view of Connolly et al. (US 2015/0067018).
Regarding claim 2, Zia and Huval do not expressly disclose further comprising: acquiring information about a working process of the worker; and estimating, on the basis of the information about the working process of the worker, a segment irrelevant to the working process in the video data as a segment not to be labeled, wherein the first display image includes an image of the segment not to be labeled.
However, Connolly teaches further comprising: acquiring information about a working process of the worker (see para. 0105, 0120, Connolly discusses acquiring technician work information); and estimating, on the basis of the information about the working process of the worker, a segment irrelevant to the working process in the video data as a segment not to be labeled, wherein the first display image includes an image of the segment not to be labeled (see para. 0120, Connolly discusses user confirm performance of an action and/or visually inspect, measure, operate, or otherwise examine component subsystem and/or equipment).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 2. The result would have been expected, routine, and predictable in order to perform work image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia and Huval in this manner in order to improve image annotation of a workplace by allowing a user to annotate images from a same time period. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia and Huval, while the teaching of Connolly continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images within a time period and annotating the objects to analyze work progress. The Zia, Huval, and Connolly systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 3, Connolly teaches wherein the information about the working process includes information included in a work procedure manual prepared for each work in the factory (see para. 0186, Connolly discusses guidance information in one or more manuals, books, checklists, user device presents documents that assist the user in performing the work scope).
The same motivation of claim 2 is applied to claim 3. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 3. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 4, Connolly teaches wherein the information about the working process includes information about factory equipment used by the worker (see para. 0186, Connolly discusses guidance information in one or more manuals, books, checklists, user device presents to the user documents that assist the user in performing the work scope).
The same motivation of claim 2 is applied to claim 4. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 4. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 5, Connolly teaches further comprising generating a second display image including an index for performance of the evaluation model, on the basis of the first information tag and the second information tag (see para. 0110, Connolly discusses causes another recommended work scope to be selected by the user; see para. 0073, Connolly discusses work scope recommends performing the listed actions to perform an inspection of an engine of the equipment; see para. 0073, Connolly discusses user interface that presents estimated completion times for various actions of a work scope).
The same motivation of claim 2 is applied to claim 5. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 5. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 6, Connolly teaches wherein the index for the performance of the evaluation model includes an index indicating precision of the evaluation model (see para. 0082-0083, Connolly discusses indicating estimated completion times, and resources to produce results).
The same motivation of claim 2 is applied to claim 6. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 6. The result would have been expected, routine, and predictable in order to perform work image annotation.
Claims 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980) in view of Connolly et al. (US 2015/0067018) in view of Itoh et al. (US 2022/0012514) (WO 2020/0194961A1) published on 10/1/2020.
Regarding claim 7, Zia, Huval, and Connolly do not expressly disclose wherein the evaluation model is obtained by learning using a set of another video data obtained by capturing a video of the worker and a third information tag as teacher data, the third information tag being set by the user in a segment including the another video data.
However, Itoh teaches wherein the evaluation model is obtained by learning using a set of another video data obtained by capturing a video of the worker and a third information tag as teacher data, the third information tag being set by the user in a segment including the another video data (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and identification information may be corrected by the operator to perform annotation and re-annotation).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 7. The result would have been expected, routine, and predictable in order to perform image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia, Huval, and Connolly in this manner in order to improve image annotation of a workplace by allowing a user to annotate images by implementing a trained model using target image data to properly identify objects. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia, Huval, and Connolly, while the teaching of Itoh continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images and annotating the objects to analyze work progress using a trained model with reference images that corrects errors to prevent improper classifications. The Zia, Huval, Connolly, and Itoh systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 8, Itoh teaches wherein the evaluation model is relearned when the user corrects the video data or the second information tag in display of the first display image (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and operator input to correct image annotation).
The same motivation of claim 7 is applied to claim 8. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 8. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 9, Itoh teaches wherein the evaluation model is obtained by learning using a set of a target area and the third information tag as teacher data, the target area being specified by the user in another video data obtained by capturing a video of the worker (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and operator input to correct image annotation).
The same motivation of claim 7 is applied to claim 9. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 9. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 10, Itoh teaches further comprising generating, for the index for the performance of the evaluation model, a third display image that indicates a difference between the second information tag attached by the user and the first information tag attached by the evaluation model (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, each annotation module may assign different information).
The same motivation of claim 7 is applied to claim 10. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 10. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 11, Itoh teaches wherein the evaluation model is obtained by learning of at least two pieces of the teacher data (see figure 4, figures 8-12, para. 0049, Itoh discusses a learning model updated with various image data).
The same motivation of claim 7 is applied to claim 11. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 11. The result would have been expected, routine, and predictable in order to perform work image annotation.
Conclusion
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNY A CESE whose telephone number is (571) 270-1896. The examiner can normally be reached on Monday – Friday, 9am – 4pm.
If attempts to reach the primary examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kenny A Cese/
Primary Examiner, Art Unit 2663
Prosecution Insights