DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed on 7/24/2026 regarding claims 1, 12, 13 has been fully considered. The highlighted arguments are listed below and will be addressed accordingly.
Argument #1 (REMARKS/ARGUMENTS, page 8): Applicant asserts that claims 1, 12, 13 overcomes the rejection under 35 U.S.C. 103 because,
“The Office Action relies on Zia ( e.g., Figs. 4 and 5) to teach the first and second display areas extending along a common time axis. However, Zia merely discloses a user interface that lists different operational tasks or sub-actions (e.g., "Pick up CPU," "Close cover") generated by neural network model on separate tracks or rows along a timeline. Zia does not teach or suggest a comparison display where a first display area (corresponding to an evaluation model tag) and a second display area (corresponding to a manual tag) extend adjacent to each other along a common time axis to indicate the first information tag and the second information tag in contrast The purpose of Zia's interface is to monitor a sequence of different actions, not to contrast predicted machine learning tags with human-annotated tags on adjacent tracks.”
Examiner respectfully disagrees, and indicates that the cited Prior Art reasonably address limitations of the claimed invention. Applicant is reminded that Examiner will interpret each claim in the broadest reasonable sense, as such, the claims and only the claims form the metes and bounds of the invention.
The applicant has failed to produce evidence that the prior art combination of Zia and Huval does not carry out the steps in claims 1, 12, and 13. Zia does suggest the noted features by disclosing, in figure 4 and figure 5, first display area showing a task tagged and the second display area showing a different task tagged extending adjacent to each other along a common time axis corresponding to the timeline of the video data to indicate the first information tag and the second information tag in contrast on the common time axis. In figure 4 and figure 5, Zia teaches a user interface display that contains annotated tasks that are captured in videos of workers, the tasks are displayed in contrast with different time segments on a common timeline. The Examiner suggests the Applicant further amend the language in this claim limitation to overcome Zia and place the claims in condition for allowance.
Argument #2 (REMARKS/ARGUMENTS, page 9): Applicant asserts that claims 1, 12, 13 overcomes the rejection under 35 U.S.C. 103 because,
“However, Huval fails to remedy the deficiencies of Zia. While Huval teaches an interface for a human to manually correct an automatically generated label, Huval does NOT teach a comparison display where the automatic label and the manual label are maintained in distinct display areas extending adjacent to each other on a timeline to show them in contrast In Huval's workflow, a manual correction typically overwrites, modifies, or confirms the automated label in the same view, rather than displaying them side-by-side or adjacently along a timeline for contrasting purposes.”
Examiner respectfully disagrees, and indicates that the cited Prior Art reasonably address limitations of the claimed invention. Applicant is reminded that Examiner will interpret each claim in the broadest reasonable sense, as such, the claims and only the claims form the metes and bounds of the invention.
The applicant has failed to produce evidence that the prior art combination of Zia and Huval does not carry out the steps in claims 1, 12, and 13. In col. 20 lines 37-40, Huval teaches an input/output display for the video data. Also, in col. 20 lines 60-67, Huval teaches a touch screen display device to gather user input. In figure 1, figure 3, figure 4, and paragraph 0008, Huval teaches manual label annotation and automatic label annotation using a neural network at the same time T1 on video data.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 12, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980).
Regarding claim 1, Zia teaches information processing method performed by a computer, the method comprising: attaching a first information tag to a target segment of video data on a basis of the video data obtained by capturing a video of a worker using an evaluation model according to a machine learning technique, the first information tag relating to a factory work (see figure 3, figure 4, figure 7, figure 8, Zia discusses annotating tasks captured in video of a worker using a trained machine learning model); and
wherein the first display image includes a comparison display including a first display area corresponding to the first information tag and a second display area corresponding to the second information tag, the first display area and the second display area extending adjacent to each other along a common time axis corresponding to the time line of the video data to indicate the first information tag and the second information tag in contrast on the common time axis, the first display area including an indication of a segment of the first information tag at a position corresponding to a time segment on the common time axis, and the second display area including an indication of a segment of the second information tag at a position corresponding to a time segment on the common time axis (see figure 4, figure 5, Zia discusses a user interface display that contains annotated tasks that are captured in videos of workers, the tasks are displayed in contrast with different time segments on a common timeline).
Huval teaches generating a first display image that shows both a second information tag and the first information tag simultaneously on the same time line of the video data (see col. 20 lines 37-40, Huval discusses an input/output display for the video data; see in col. 20 lines 60-67, Huval discusses a touch screen display device to gather user input), the second information tag relating to the factory work and being attached to a segment of the video data by a user manually without using the evaluation model, the first information tag being attached to the target segment using the evaluation model according to the machine learning technique (see figure 1, figure 3, figure 4, para. 0008, Huval discusses manual label annotation and automatic label annotation using a neural network at the same time T1).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia with Huval to derive at the invention of claim 1. The result would have been expected, routine, and predictable in order to perform work image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia in this manner in order to improve image annotation of a workplace by allowing a user to annotate images from a same time period. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia, while the teaching of Huval continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images within a time period and annotating the objects to analyze work progress. The Zia and Huval systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claim 12 is rejected as applied to claim 1 as pertaining to a corresponding device.
Claim 13 is rejected as applied to claim 1 as pertaining to a corresponding non-transitory computer readable medium.
Claims 2-6 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980) in view of Connolly et al. (US 2015/0067018).
Regarding claim 2, Zia and Huval do not expressly disclose further comprising: acquiring information about a working process of the worker; and estimating, on the basis of the information about the working process of the worker, a segment irrelevant to the working process in the video data as a segment not to be labeled, wherein the first display image includes an image of the segment not to be labeled.
However, Connolly teaches further comprising: acquiring information about a working process of the worker (see para. 0105, 0120, Connolly discusses acquiring technician work information); and estimating, on the basis of the information about the working process of the worker, a segment irrelevant to the working process in the video data as a segment not to be labeled, wherein the first display image includes an image of the segment not to be labeled (see para. 0120, Connolly discusses user confirm performance of an action and/or visually inspect, measure, operate, or otherwise examine component subsystem and/or equipment).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 2. The result would have been expected, routine, and predictable in order to perform work image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia and Huval in this manner in order to improve image annotation of a workplace by allowing a user to annotate images from a same time period. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia and Huval, while the teaching of Connolly continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images within a time period and annotating the objects to analyze work progress. The Zia, Huval, and Connolly systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 3, Connolly teaches wherein the information about the working process includes information included in a work procedure manual prepared for each work in the factory (see para. 0186, Connolly discusses guidance information in one or more manuals, books, checklists, user device presents documents that assist the user in performing the work scope).
The same motivation of claim 2 is applied to claim 3. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 3. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 4, Connolly teaches wherein the information about the working process includes information about factory equipment used by the worker (see para. 0186, Connolly discusses guidance information in one or more manuals, books, checklists, user device presents to the user documents that assist the user in performing the work scope).
The same motivation of claim 2 is applied to claim 4. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 4. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 5, Connolly teaches further comprising generating a second display image including an index for performance of the evaluation model, on the basis of the first information tag and the second information tag (see para. 0110, Connolly discusses causes another recommended work scope to be selected by the user; see para. 0073, Connolly discusses work scope recommends performing the listed actions to perform an inspection of an engine of the equipment; see para. 0073, Connolly discusses user interface that presents estimated completion times for various actions of a work scope).
The same motivation of claim 2 is applied to claim 5. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 5. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 6, Connolly teaches wherein the index for the performance of the evaluation model includes an index indicating precision of the evaluation model (see para. 0082-0083, Connolly discusses indicating estimated completion times, and resources to produce results).
The same motivation of claim 2 is applied to claim 6. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia and Huval with Connolly to derive at the invention of claim 6. The result would have been expected, routine, and predictable in order to perform work image annotation.
Claims 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Zia et al. (US 11,216,656) in view of Huval (US 2018/0373980) in view of Connolly et al. (US 2015/0067018) in view of Itoh et al. (US 2022/0012514) (WO 2020/0194961A1) published on 10/1/2020.
Regarding claim 7, Zia, Huval, and Connolly do not expressly disclose wherein the evaluation model is obtained by learning using a set of another video data obtained by capturing a video of the worker and a third information tag as teacher data, the third information tag being set by the user in a segment including the another video data.
However, Itoh teaches wherein the evaluation model is obtained by learning using a set of another video data obtained by capturing a video of the worker and a third information tag as teacher data, the third information tag being set by the user in a segment including the another video data (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and identification information may be corrected by the operator to perform annotation and re-annotation).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 7. The result would have been expected, routine, and predictable in order to perform image annotation.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Zia, Huval, and Connolly in this manner in order to improve image annotation of a workplace by allowing a user to annotate images by implementing a trained model using target image data to properly identify objects. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Zia, Huval, and Connolly, while the teaching of Itoh continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of tagging different objects in various images of workplace by capturing multiple images and annotating the objects to analyze work progress using a trained model with reference images that corrects errors to prevent improper classifications. The Zia, Huval, Connolly, and Itoh systems perform image annotation, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 8, Itoh teaches wherein the evaluation model is relearned when the user corrects the video data or the second information tag in display of the first display image (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and operator input to correct image annotation).
The same motivation of claim 7 is applied to claim 8. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 8. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 9, Itoh teaches wherein the evaluation model is obtained by learning using a set of a target area and the third information tag as teacher data, the target area being specified by the user in another video data obtained by capturing a video of the worker (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, and operator input to correct image annotation).
The same motivation of claim 7 is applied to claim 9. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 9. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 10, Itoh teaches further comprising generating, for the index for the performance of the evaluation model, a third display image that indicates a difference between the second information tag attached by the user and the first information tag attached by the evaluation model (see figure 4, figures 8-12, para. 0022, 0070, 0097, Itoh discusses multiple annotation modules and learning model, each annotation module may assign different information).
The same motivation of claim 7 is applied to claim 10. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 10. The result would have been expected, routine, and predictable in order to perform work image annotation.
Regarding claim 11, Itoh teaches wherein the evaluation model is obtained by learning of at least two pieces of the teacher data (see figure 4, figures 8-12, para. 0049, Itoh discusses a learning model updated with various image data).
The same motivation of claim 7 is applied to claim 11. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Zia, Huval, and Connolly with Itoh to derive at the invention of claim 11. The result would have been expected, routine, and predictable in order to perform work image annotation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNY A CESE whose telephone number is (571) 270-1896. The examiner can normally be reached on Monday – Friday, 9am – 4pm.
If attempts to reach the primary examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kenny A Cese/
Primary Examiner, Art Unit 2663
