Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status of the Application
The following is a Final Office Action.
In response to Examiner's communication of 2/10/2026, Applicant responded on 5/7/2026. Amended claim 1, 2, 4, 6, 7.
Claims 1-7 are pending in this application and have been examined.
Response to Amendment
Applicant's amendments to claims 4 are not sufficient to overcome the claim objection set forth in the previous action. The claim objection is hereby withdrawn.
Applicant's amendments to claims 1, 2, 4, 6, 7 are not sufficient to overcome the 35 USC 101 rejections set forth in the previous action.
Applicant's amendments to claims 1, 2, 4, 6, 7 are not sufficient to overcome the prior art rejections set forth in the previous action.
Response to Arguments – 35 USC § 101
Applicant’s arguments with respect to the rejections have been fully considered, but they are not persuasive.
Applicant submits, “…As amended, independent claim 1 (and similarly claim 7) is directed to a specific machine- implemented video analysis and visualization technique for generating synchronized work- performance analysis display information, not to an abstract idea. Even assuming, arguendo, that the claims recite some abstract concept, the claims integrate that concept into a practical application through specific limitations that require recognition of target objects and worker actions from video recordings, generation of a display screen having both a video image indicator and an analysis result indicator arranged and synchronized in a particular manner, and display of a timeline cursor in response to playback operations. The claims therefore are patent- eligible under Step 2A, and, in any event, include significantly more than any alleged judicial exception under Step 2B.…The claims recite a work analyzing device having a processor configured to process video recordings of a worker's actions, recognize one or more target objects to be handled by the worker, determine types of the target objects, recognize one or more actions of the worker forming a work, determine types of the actions, and generate a display screen including a video image indicator and an analysis result indicator. These are machine-implemented operations performed on video data and temporal event information, not purely mental steps that a human realistically performs in their mind or with pen and paper. The claim requires coordinated video processing, object/action recognition, and time-ordered visualization that is properly understood as a technological process for analyzing recorded work activity. The claims also are not directed to a method of organizing human activity in the sense used in the eligibility jurisprudence. The claims do not recite rules for managing workers, assigning tasks, scheduling labor, or conducting business operations. Instead, the claims are directed to a technical system that analyzes video recordings and generates a specialized, synchronized display of detected worker actions relative to target objects over time. Any human review of the output is merely a downstream use of the generated information and is not what is claimed. Accordingly, the claims are not properly characterized as being directed to organizing human activity…The amended claims recite a specific display architecture in which the processor generates a display screen including: a video image indicator displaying a work video image for a designated period of time; an analysis result indicator displaying analysis result information for the designated period of time; action indicators arranged vertically in a row for respective types of target objects; each action indicator extending along a horizontal time axis; and one or more action images arranged in time order, each action image representing a type of action and a period of time during which the action is performed. In response to a user's operation for playback of the work video image, the system displays a timeline cursor indicating a current playback position on the analysis result indicator. This is not merely generic output of information. It is a concrete, synchronized user interface that links raw video playback to a structured time-series representation of detected object-action events. This arrangement improves the functioning of the work-analysis system itself by enabling the user to correlate the worker's actions with the corresponding analysis result at the exact playback position. Conventional systems may present a video and a separate chart, but the claimed invention requires a coordinated display in which the analysis result indicator is organized by target object type, aligned to a horizontal time axis, and synchronized with playback through a timeline cursor. That specific configuration reduces the need for manual correlation between video and analysis output and provides a practical technological improvement in the way work-performance status is reviewed and understood. The claims therefore are integrated into a practical application and are not "directed to" an abstract idea…The Examiner has not shown that the additional claim elements, individually or in combination, are well-understood, routine, and conventional. The Office Action identifies generic components such as a processor, camera, recorder, storage, and terminal, but those components are not the point of novelty in the amended claims. The relevant inquiry is whether the claimed combination-video-based recognition of target objects and worker actions, construction of a vertically arranged analysis result indicator by target object type, display of action images along a horizontal time axis, and synchronized cursor-based playback-is conventional. The Office Action does not provide evidence that this particular combination of processing and display operations was routine in the art. A bare assertion that the claims use generic computer components is insufficient to establish a Step 2B deficiency. Moreover, the claimed limitations amount to significantly more than simply applying an abstract idea on a generic computer. The claims recite a specialized processing and display workflow that transforms raw work video into a structured analysis screen with coordinated visual elements dynamically updated during playback. The vertical arrangement of action indicators by target object type, the horizontal time-axis display of action images, and the synchronized timeline cursor collectively provide a specific mechanism for understanding worker actions over time. These features are not incidental post-solution activity or mere data presentation; they are central to the claimed invention and provide an inventive concept that places the claims well beyond any alleged abstract idea…” The Examiner respectfully disagrees.
The claims and the argued elements, are recite and directed to, ….work-analysis…recognize one or more target objects to be handled by the worker, determine types of the target objects, recognize one or more actions of the worker forming a work, determine types of the actions…detected worker actions relative to target objects over time… improvement in the way work-performance status is reviewed and understood…, is a problem reciting and directing to mental process (i.e. human watching human workers performing work actions and managing human worker performing work actions), organizing human activities (i.e. human watching human workers performing work actions and managing human worker performing work actions), as established in Step 2A Prong 1. This problem does not specifically arise in the realm of computer technology, but rather, this problem existed and was addressed long before the advent of computers. Thus, the claims do not recite a technical improvement to a technical problem. Additionally, pursuant to the broadest reasonable interpretation, as an ordered combination, each of the additional elements are computing elements recited at high level of generality implementing the abstract idea, and thus, are no more than applying the abstract idea with generic computer components, i.e. computer, video playback, display screen. Further, these additional elements generally link the abstract idea to a technical environment, namely the environment of a computer and display screen, performing extra solution activities. Therefore, as a whole, the additional elements do not integrate the abstract ideas into a practical application in Step 2A Prong 2 (apply it and general link) or amount to significantly more in Step 2B (apply it and WURC).
Even novel and newly discovered judicial exceptions are still exceptions, despite their novelty. July 2015 Update, p. 3; see SAP America Inc. v. Investpic, LLC, No. 2017-2081, slip op. at 2 (Fed Cir. May 15, 2018).
Simply reciting specific limitations that narrow the abstract idea does not make an abstract idea non-abstract. 79 Fed. Reg. 74631; buySAFE Inc. v. Google, Inc., 765 F.3d 1350, 1355 (2014); see SAP America at p. 12. As discussed in SAP America, no matter how much of an advance the claims recite, when “the advance lies entirely in the realm of abstract ideas, with no plausibly alleged innovation in the non-abstract application realm,” “[a]n advance of that nature is ineligible for patenting.” Id. at p. 3.
Claims can recite a mental process even if they are claimed as being performed on a computer. The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The Court concluded that the algorithm could be performed purely mentally even though the claimed procedures “can be carried out in existing computers long in use, no new machinery being necessary.” 409 U.S at 67, 175 USPQ at 675. See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699 (concluding that concept of “anonymous loan shopping” recited in a computer system claim is an abstract idea because it could be “performed by humans without a computer”).
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, “claiming the improved speed or efficiency inherent with applying the abstract idea on a computer” does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).
TLI Communications provides an example of a claim invoking computers and other machinery merely as a tool to perform an existing process. The court stated that the claims describe steps of recording, administration and archiving of digital images, and found them to be directed to the abstract idea of classifying and storing digital images in an organized manner. 823 F.3d at 612, 118 USPQ2d at 1747. The court then turned to the additional elements of performing these functions using a telephone unit and a server and noted that these elements were being used in their ordinary capacity (i.e., the telephone unit is used to make calls and operate as a digital camera including compressing images and transmitting those images, and the server simply receives data, extracts classification information from the received data, and stores the digital images based on the extracted information). 823 F.3d at 612-13, 118 USPQ2d at 1747-48. In other words, the claims invoked the telephone unit and server merely as tools to execute the abstract idea. Thus, the court found that the additional elements did not add significantly more to the abstract idea because they were simply applying the abstract idea on a telephone network without any recitation of details of how to carry out the abstract idea.
[E]xamples where the courts have found the additional elements to be mere instructions to apply an exception, because they do no more than merely invoke computers or machinery as a tool to perform an existing process include:
i. A commonplace business method or mathematical algorithm being applied on a general purpose computer, Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014); Gottschalk v. Benson, 409 U.S. 63, 64, 175 USPQ 673, 674 (1972); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015);
ii. Generating a second menu from a first menu and sending the second menu to another location as performed by generic computer components, Apple, Inc. v. Ameranth, Inc., 842 F.3d 1229, 1243-44, 120 USPQ2d 1844, 1855-57 (Fed. Cir. 2016);
iii. A process for monitoring audit log data that is executed on a general-purpose computer where the increased speed in the process comes solely from the capabilities of the general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016);
iv. A method of using advertising as an exchange or currency being applied or implemented on the Internet, Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 715, 112 USPQ2d 1750, 1754 (Fed. Cir. 2014);
v. Requiring the use of software to tailor information and provide it to the user on a generic computer, Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1370-71, 115 USPQ2d 1636, 1642 (Fed. Cir. 2015);
Examples that the courts have indicated may not be sufficient to show an improvement in computer-functionality:
i. Generating restaurant menus with functionally claimed features, Ameranth, 842 F.3d at 1245, 120 USPQ2d at 1857;
ii. Accelerating a process of analyzing audit log data when the increased speed comes solely from the capabilities of a general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016);
iii. Mere automation of manual processes, such as using a generic computer to process an application for financing a purchase, Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044, 1055, 123 USPQ2d 1100, 1108-09 (Fed. Cir. 2017) or speeding up a loan-application process by enabling borrowers to avoid physically going to or calling each lender and filling out a loan application, LendingTree, LLC v. Zillow, Inc., 656 Fed. App'x 991, 996-97 (Fed. Cir. 2016) (non-precedential);
iv. Recording, transmitting, and archiving digital images by use of conventional or generic technology in a nascent but well-known environment, without any assertion that the invention reflects an inventive solution to any problem presented by combining a camera and a cellular telephone, TLI Communications, 823 F.3d at 611-12, 118 USPQ2d at 1747;
vi. Instructions to display two sets of information on a computer display in a non-interfering manner, without any limitations specifying how to achieve the desired result, Interval Licensing LLC v. AOL, Inc., 896 F.3d 1335, 1344-45, 127 USPQ2d 1553, 1559-60 (Fed. Cir. 2018);
vii. Providing historical usage information to users while they are inputting data, in order to improve the quality and organization of information added to a database, because “an improvement to the information stored by a database is not equivalent to an improvement in the database’s functionality,” BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281, 1287-88, 127 USPQ2d 1688, 1693-94 (Fed. Cir. 2018); and
viii. Arranging transactional information on a graphical user interface in a manner that assists traders in processing information more quickly, Trading Technologies v. IBG LLC, 921 F.3d 1084, 1093-94, 2019 USPQ2d 138290 (Fed. Cir. 2019).
Response to Arguments – Prior Art
Applicant’s arguments with respect to the rejections have been fully considered, but they are not persuasive. However, Applicant’s remarks are moot in light of new grounds of rejections necessitated by Applicant’s amendments.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 1 (similarly 7) recite, “A… to perform operations to output analysis result display information on a work performance status of a worker based on video recordings of actions of the worker,
wherein the … is configured to:
based on the video recordings, recognize one or more target objects to be handled by the worker and determine types of the target objects, while recognizing one or more actions of the worker, the actions forming a work, and determining types of the actions; and
generate a display … including:
… showing the worker's actions for a designated period of time; and
an analysis result indicator that displays the analysis result display information on the worker's actions during the designated period of time, wherein the analysis result display information includes action indicators arranged vertically in a row for respective types of the target objects, each action indicator for a corresponding target object extending along a horizontal time axis and including one or more action … arranged in order of time, each action … representing a type of the action and a period of time during which the action is performed, and
…. “
Analyzing under Step 2A, Prong 1:
The limitations regarding, …based on the video recordings, recognize one or more target objects to be handled by the worker and determine types of the target objects, while recognizing one or more actions of the worker, the actions forming a work, and determining types of the actions; and generate a display … including: … showing the worker's actions for a designated period of time; and an analysis result indicator that displays the analysis result display information on the worker's actions during the designated period of time, wherein the analysis result display information includes action indicators arranged vertically in a row for respective types of the target objects, each action indicator for a corresponding target object extending along a horizontal time axis and including one or more action … arranged in order of time, each action… representing a type of the action and a period of time during which the action is performed, and…, under the broadest reasonable interpretation, can include a human using their mind and using pen and paper to perform the above identified limitations, therefore, the claims recite a mental process.
Further, …based on the video recordings, recognize one or more target objects to be handled by the worker and determine types of the target objects, while recognizing one or more actions of the worker, the actions forming a work, and determining types of the actions; and generate a display … including: … showing the worker's actions for a designated period of time; and an analysis result indicator that displays the analysis result display information on the worker's actions during the designated period of time, wherein the analysis result display information includes action indicators arranged vertically in a row for respective types of the target objects, each action indicator for a corresponding target object extending along a horizontal time axis and including one or more action … arranged in order of time, each action… representing a type of the action and a period of time during which the action is performed, and…, are managing human worker performing work actions, which are managing interactions between people, therefore the claims recite certain methods of organizing human activities.
Accordingly, the claims are directed to a mental process, certain methods of organizing human activities, and thus, the claims are directed to an abstract idea under the first prong of Step 2A.
Analyzing under Step 2A, Prong 2:
This judicial exception is not integrated into a practical application under the second prong of Step 2A.
In particular, the claims recite the additional elements beyond the recited abstract idea identified under Step 2A, Prong 1, such as:
Claim 1, 7: work analyzing device in which a processor is caused, processor, display screen, a video image indicator that displays a work video image, action images, in response to a user's operation for playback of the work video image, in response to a user's operation for playback of the work video image displayed on the video image indicator, display a timeline cursor indicating a current playback position on the analysis result indicator
Claim 2: outputs the video recordings as a video image that allows for playback
, and pursuant to the broadest reasonable interpretation, as an ordered combination, each of the additional elements are computing elements recited at high level of generality implementing the abstract idea, and thus, are no more than applying the abstract idea with generic computer components.
Further, these additional elements generally link the abstract idea to a technical environment, namely the environment of a computer.
Additionally, with respect to, “…based on the video recordings, recognize…”, “…identifies…”, “…detecting…”, “…output analysis result display information…”, “…showing…”, “…displays the analysis result display information…”, these elements do not add a meaningful limitations to integrate the abstract idea into a practical application because they are extra-solution activity, pre and post solution activity - i.e. data gathering – “…based on the video recordings, recognize…”, “…identifies…”, “…detecting…”, data output – “…output analysis result display information…”
Analyzing under Step 2B:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B.
As noted above, the aforementioned additional elements beyond the recited abstract idea are not sufficient to amount to significantly more than the recited abstract idea because, as an order combination, the additional elements are no more than mere instructions to implement the idea using generic computer components (i.e. apply it).
Additionally, as an order combination, the additional elements append the recited abstract idea to well-understood, routine, and conventional activities in the field as individually evinced by the applicant’s own disclosure, as required by the Berkheimer Memo, in at least:
[0003] Known such technologies for analyzing a work performance status of a worker include a method which involves capturing video recordings of actions of a worker with a camera; analyzing, from the video recordings, a work performance status of a worker; and presenting to a user a Gantt chart that visualizes a status of the worker's actions in each process step and/or a video image of the working worker during work (Patent Document 1).
[0026] Figure 1 is a diagram showing an overall configuration of a work analyzing system according to an embodiment of the present disclosure.
[0027] The work analyzing system is configured to analyze a work performance status of a worker in a workplace such as a factory to thereby present a result of analysis to a user. The system includes a camera 1, a recorder 2, a server 3 (work analyzing device), and an administrator terminal 4.
[0028] The camera 1 images a worker working at the worker's place.
[0029] The recorder 2 records video data provided from the camera 1.
[0030] The server 3 acquires video recordings from the camera 1 and/or the recorder 2, analyzes a work performance status of a worker based on video recordings, and outputs a result of the analysis. The server 3 is installed in a facility (e.g., a factory) and performs work analysis for the facility. The server 3 may also be configured as a cloud computer and perform work analysis for a plurality of facilities.
[0031] The administrator terminal 4 is used by a system administrator or a work administrator (user), and is implemented by a PC, a tablet terminal, or any other suitable device. The system administrator uses the administrator terminal 4 to perform settings for various operations performed by the server 3. In addition, the administrator terminal 4 displays results of analysis provided from the server 3 so that the work administrator can view the results.
[0032] Next, a schematic configuration of the server 3 will be described. Figure 2 is a block
diagram showing a schematic configuration of the server 3.
[0033] The server 3 includes a video input device 11, a screen output device 12, a storage 13, and a processor 14.
[0034] The video input device 11 receives at video recordings recorded in the recorder 2.
[0035] The screen output device 12 outputs data of an analysis result display screen generated by the processor 14, and the analysis result display screen (Figure 6) is displayed on the administrator terminal 4.
[0036] The storage 13 stores programs which are executable by a processor 14 and other data. The storage 13 stores video recordings for analysis acquired from the recorder 2. The storage 13 stores an object action recognition model (object action estimation model) to be used by the processor 14, as well as an analysis result video and analysis result information generated by the processor 14.
[0037] The processor 14 performs various operations by executing programs stored in the storage 13. In the present embodiment, the processor 14 performs an analysis operation P1, an analysis result visualization operation P2, and other operations.
[0038] In the analysis operation P1, the processor 14 makes analysis of a work performance status of a worker based on video recordings of actions of the worker to generate analysis result information.
[0039] In the analysis result visualization operation P2, the processor 14 visualizes the analysis result information generated in the analysis operation P1 and presents the visualized information to a user. In the present embodiment, the processor 14 generates an analysis result display screen (Figure 6).
[0040] Next, the analysis operation P1 performed by the server 3 will be described. Figure 3 is a block diagram showing an outline of the analysis operation P1 performed by the server 3.
[0041] In the server 3, the processor 14 performs an analysis operation P1. The analysis operation P1 includes a video acquisition operation P11, an object action recognition operation P12, an analysis result video generation operation P13, a work cycle detection operation P14, an abnormality detection operation P15, and an output operation P16.
[0042] In the video acquisition operation P11, the processor 14 acquires video recordings to be analyzed from the storage 13.
[0089] While specific embodiments of the present disclosure are described herein for illustrative purposes, the present disclosure is not limited to those specific embodiments. Various changes, substitutions, additions, and omissions may be made to elements of the embodiments without departing from the scope of the present disclosure. Moreover, elements and features of the different embodiments may be combined with each other to yield another embodiment of the present disclosure.
Furthermore, as an ordered combination, these elements amount to generic computer components receiving or transmitting data over a network, performing repetitive calculations, electronic record keeping, and storing and retrieving information in memory, which, as held by the courts, are well-understood, routine, and conventional. See MPEP 2106.05(d).
Moreover, the remaining elements of dependent claims do not transform the recited abstract idea into a patent eligible invention because these remaining elements merely recite further abstract limitations that provide nothing more than simply a narrowing of the abstract idea recited in the independent claims.
Looking at these limitations as an ordered combination adds nothing additional that is sufficient to amount to significantly more than the recited abstract idea because they simply provide instructions to use a generic arrangement of generic computer components to “apply” the recited abstract idea, perform insignificant extra-solution activity, and generally link the abstract idea to a technical environment. Thus, the elements of the claims, considered both individually and as an ordered combination, are not sufficient to ensure that the claim as a whole amounts to significantly more than the abstract idea itself. Since there are no limitations in these claims that transform the exception into a patent eligible application such that these claims amount to significantly more than the exception itself, claims 1-7 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art.
Ascertaining the differences between the prior art and the claims at issue.
Resolving the level of ordinary skill in the pertinent art.
Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 is/are rejected under 35 U.S.C. 103 as being unpatentable by JP Patent Publication to JP2019016226A to Kudo et al., (hereinafter referred to as “Kudo”) in view of US Patent Publication to US20210125518A1 to Wada et al., (hereinafter referred to as “Wada”)
As per Claim 1, Kudo teaches: A work analyzing device in which a processor is caused to perform operations to output analysis result display information on a work performance status of a worker based on video recordings of actions of the worker,
wherein the processor is configured to: ([0011]-[0019])
based on the video recordings, recognize one or more target objects to be handled by the worker and determine types of the target objects, while recognizing one or more actions of the worker, the actions forming a work, and determining types of the actions; and (in at least [0011] The work data management system W shown in FIG. 1 is a system for managing data (referred to as work data) relating to work at a work site. In the present embodiment, in order to manufacture a prescribed product, a state in which an operator repeatedly performs a predetermined operation (processing of a member, etc.) is photographed and analysis and display of work are performed based on the imaging result There. The above-mentioned member may be substantially the same as the finished product, or it may be a component of a finished product. [0032] For example, in the detailed process A1 shown in FIG. 4, the process determining unit 16c (see FIG. 2) performs the following process. That is, when the large step γ is "processing" and the event E 2 occurs within 10 seconds after the occurrence of the event E 1, and the state satisfying both of these continues for more than 1 second, the process determining means 16 c , It is determined that "the worker is setting a member in the processing machine". As described above, the detailed process classifying means 16 (see FIG. 2) classifies the work performed by the worker at the work site into a plurality of time-series detailed processes based on the work data including the image data of the work site. Further, the detailed process classifying means 16 specifies the detailed process performed by the operator based on the change in the position of the predetermined part (head and hand) on the body of the worker. [0070] the large process performed by the worker is subdivided into a plurality of time-series detailed processes on the basis of the video data of the work site and the like as the Gantt chart P1 to the result display means 26 (See FIG. 6A). As a result, the user (administrator) can easily grasp how long it took in each detailed process. In addition, it is unnecessary for an operator to attach a position sensor (not shown) or an acceleration sensor (not shown), thereby saving labor on the worker side. [0093] For example, the information processing apparatus 100 ⟨/ b⟩ Ba analyzes the work of processing one product repeatedly by a worker, and the information processing apparatus 100 ⟨/ b⟩ Bb analyzes the work of another worker repeating the assembly of the product after the processing There. That is, when a plurality of types of work are sequentially performed, when a predetermined product is manufactured, each of the plurality of types of work is a large process including a plurality of time-series detailed processes, and different workers It is in charge. In the present embodiment, the information processing apparatuses 100 Ba and 100 Bb individually analyze the operations of a plurality of workers performing such flow work (processing and assembly), and the analysis results are integrated by the data integration apparatus 200 ing. [0094] The work site as the measurement target (imaging target) of the first measurement device G 1 and the second measurement device G 2, the work site as the measurement target (imaging target) of the first measurement device G 1 and the second measurement device G 2, Shall not be far apart.)
generate a display screen including: ([0044])
a video image indicator that displays a work video image showing the worker's actions for a designated period of time; and (in at least [0045] ⟨Example of Screen Image⟩ FIG. 6A is a screen image when the reproduction time bar P 5 is located in the detailed step A 1 (see FIG. 1 as appropriate). The screen image shown in FIG. 6A includes a Gantt chart P 1, a plurality of box beard images P 2 superimposed on the Gantt chart P 1, and measurement data display areas P 3 and P 4.)
an analysis result indicator that displays the analysis result display information on the worker's actions during the designated period of time, wherein the analysis result display information includes action indicators arranged vertically in a row for respective types of the …, each action indicator for a corresponding target object extending along a horizontal time axis and including one or more action images arranged in order of time, each action image representing a type of the action and a period of time during which the action is performed, and (in at least [0053] At the playback time shown in FIG. 6A, since the work is normally performed as described above, the head position marker P 8 exists in the existence probability distribution P 9 and the hand position marker P 10 exists in the existence probability distribution P 11 It is present in. In this way, while comparing the Gantt chart P1 and the box beard figure P2 with the existence probability distributions P9 and P11, the user (administrator) can visually grasp what kind of action the worker performed it can. [0054] FIG. 6B shows a screen image when the reproduction time bar P 5 is positioned in the detailed step M 1. In the example shown in FIG. 6B, the work deviation marker P 6 is displayed in the row of the chart of the detailed process M 1. The work deviation marker P 6 is a marker displayed on the element Pa 1 of the Gantt chart P 1 when the operation of the operator deviates from the work repeated in the past. In this manner, the data visualizing means 24, in a predetermined detailed process, when the position of a predetermined site (the head or the hand of the operator) deviates from the existence probability distribution (the existence probability is substantially zero), corresponds to this detailed process The work deviation marker P 6 is displayed on the element Pa 1 of the Gantt chart P 1. [0055] When a work deviation occurs, the data visualizing means 24 preferentially displays images on the side deviating from the existence probability distribution in the measurement data display areas P 3 and P 4, among the images of the head and hand of the operator. In the example shown in FIG. 6B, the position marker P 8 of the head and the existence probability distribution P 9 of the head are displayed in a superimposed manner in the measurement data display area P 3 together with the image of the distant view (the shooting result of the work site). In addition, in the measurement data display area P 4, the position marker P 8 of the head and the flow line history P 12 of the head are superimposed and displayed together with the image of the distant view. The flow line history P 12 is a flow line history of the position of the head in the detailed process M 1. In this way, by displaying the flow line history P 12 in addition to the Gantt chart P 1 and the like, the user can efficiently confirm the information on the work deviation. [0056] FIG. 6C shows a screen image when the reproduction time bar P 5 is positioned in the detailed step A 2. In the example shown in FIG. 6C, a flow line variation abnormality marker P 7 is displayed in the row of the chart of the detailed process A 2. The flow line variation abnormality marker P 7 is a marker displayed in association with the element Pa 1 of the Gantt chart P 1 when the operation (ie, flow line history) of the work repeated in the past largely fluctuates. Whether or not to display the flow line variation abnormality marker P 7 is determined based on the variance value of the flow line history P 12. [0057] As described above, when the variance value of a plurality of flow line histories is equal to or larger than the predetermined threshold value in the predetermined detailed process, the data visualizing means 24 adds a flow line variation abnormality to the element Pa 1 of the Gantt chart P 1 corresponding to this detailed process The marker P 7 is displayed.[0058] In addition, when there is an abnormality in the flow line variation, the data visualizing means 24 displays the image of the side of the worker's head / hand where the variance value of the flow line history exceeds the predetermined threshold in the measurement data display areas P 3, P 4 As shown in FIG. In the example shown in FIG. 6C, since the variance value of the flow line history of the hand exceeds the predetermined threshold value, the position marker P 10 of the hand and the existence probability distribution P 11 of the hand are superimposed and displayed together with the image of the foreground in the measurement data display area P 3 There. In addition, in the measurement data display area P 4, the position marker P 10 of the hand and the flow line history P 13 of the hand are superimposed and displayed together with the image of the foreground. By displaying the flow line history P 13 in addition to the Gantt chart P 1 or the like, the user can efficiently confirm the information on the flow line variation abnormality. [0087] FIG. 11 shows a screen image generated by the data visualizing means 24A of the work data management system WA. It should be noted that illustration of the work deviation marker P 6 (see FIG. 6A) and the flow line variation abnormality marker P 7 (see FIG. 6A) shown in the first embodiment is omitted. In the example shown in FIG. 11, in addition to the Gantt chart P1 and the reproduction time bar P5, the line P21 at the plan completion time based on the large process production plan 27b (see FIG. 9) and the line P22 on the predicted completion time based on the delay data are displayed Has been done. In this manner, the data visualizing means 24 A displays the line P 5 indicating the current time in a superimposed manner on the Gantt chart P 1, and displays a line P 21 showing the preset planning completion time of the large process and a line P 22 showing the predicted completion time And superimposed on the Gantt chart P1. [0106] FIG. 16 is a screen image generated by the data visualizing means 24B. It should be noted that illustration of the work deviation marker P 6 (see FIG. 6A) and the flow line variation abnormality marker P 7 (see FIG. 6A) shown in the first embodiment is omitted. When the production number selection button P 31 is selected by the user's operation via the second input means 25 (see FIG. 13), a list of the products to be worked in the past (that is, a list of production numbers) is displayed . In the example shown in FIG. 16, a Gantt chart P1 or the like concerning the processing / assembly of the product with the production number 0001 is displayed. [0107] Then, a Gantt chart P1 or the like integrating a plurality of kinds of work (processing / assembly) for the product selected by the production number selection button P31 is displayed. In this way, the data visualizing means 24 B displays a plurality of time-series detailed processes constituting each work (machining / assembly) as the Gantt chart P 1 on the result display means 26 (see FIG. 13). [0108] ⟨Effect⟩ According to the third embodiment, the data integration apparatus 200 associates the analysis results of the information processing apparatuses 100 ⟨/ b⟩ Ba, 100 ⟨/ b⟩ Bb with the "time" and the "location" of the head and hand of the operator as keys, Integrate work data on products. As a result, even when a plurality of workers are working at different places, it is possible to display the Gantt chart P1 or the like in which the respective works are integrated on the result display means 26 (see FIG. 13). Therefore, it is possible for the user (administrator) to efficiently grasp processes and occurrence tendencies that are frequent in work delay and work error, and productivity of the product can be enhanced. [0110] In each of the embodiments, the example in which the work data management system W extracts the position of the head and hand of the worker as the feature amount has been described, but the present invention is not limited to this. For example, the position of a device or a tool (not shown) may be added. This makes it possible to further improve the determination accuracy of the detailed process. [0114] Further, in the first embodiment, the example in which the reproduction time bar P 5 (see FIG. 6A) moves to the right as the video reproduction progresses has been described, but the present invention is not limited thereto. For example, the whole Gantt chart P1 may be moved to the left side and the reproduction time bar P5 may be displayed at a position that is always easy to see. Further, in each of the embodiments, an example in which the contents of the measurement data display areas P 3, P 4 (see FIG. 6A) are images has been described, but the present invention is not limited thereto. As long as the user can grasp the situation of the detailed process efficiently, it may be a graph display of the measurement value of the sensor data or the like.)
in response to a user's operation for playback of the work video image displayed on the video image indicator, display a timeline cursor indicating a current playback position on the analysis result indicator. (in at least [0054] FIG. 6B shows a screen image when the reproduction time bar P 5 is positioned in the detailed step M 1. In the example shown in FIG. 6B, the work deviation marker P 6 is displayed in the row of the chart of the detailed process M 1. The work deviation marker P 6 is a marker displayed on the element Pa 1 of the Gantt chart P 1 when the operation of the operator deviates from the work repeated in the past. In this manner, the data visualizing means 24, in a predetermined detailed process, when the position of a predetermined site (the head or the hand of the operator) deviates from the existence probability distribution (the existence probability is substantially zero), corresponds to this detailed process The work deviation marker P 6 is displayed on the element Pa 1 of the Gantt chart P 1 [0087] FIG. 11 shows a screen image generated by the data visualizing means 24A of the work data management system WA. It should be noted that illustration of the work deviation marker P 6 (see FIG. 6A) and the flow line variation abnormality marker P 7 (see FIG. 6A) shown in the first embodiment is omitted. In the example shown in FIG. 11, in addition to the Gantt chart P1 and the reproduction time bar P5, the line P21 at the plan completion time based on the large process production plan 27b (see FIG. 9) and the line P22 on the predicted completion time based on the delay data are displayed Has been done. In this manner, the data visualizing means 24 A displays the line P 5 indicating the current time in a superimposed manner on the Gantt chart P 1, and displays a line P 21 showing the preset planning completion time of the large process and a line P 22 showing the predicted completion time And superimposed on the Gantt chart P1. [0114] Further, in the first embodiment, the example in which the reproduction time bar P 5 (see FIG. 6A) moves to the right as the video reproduction progresses has been described, but the present invention is not limited thereto. For example, the whole Gantt chart P1 may be moved to the left side and the reproduction time bar P5 may be displayed at a position that is always easy to see. Further, in each of the embodiments, an example in which the contents of the measurement data display areas P 3, P 4 (see FIG. 6A) are images has been described, but the present invention is not limited thereto. As long as the user can grasp the situation of the detailed process efficiently, it may be a graph display of the measurement value of the sensor data or the like.)
Although implied, Kudo does not expressly disclose the following limitations, which however, are taught by Wada,
… arranged vertically in a row for respective types of the target objects… (in at least [0091] FIG. 6 is a view showing an example of an input screen displayed on the display part 10 f according to the present embodiment. In the input screen shown in FIG. 6, the operation status data 12 a for each predetermined part (herein, each of the right hand and the left hand) are displayed on a time axis. Through the operation of the input part 10 e by the user, an arbitrary time range is selected from the operation status data 12 a displayed on the time axis. When the time range selection button is pressed through the operation of the input part 10 e by the user, the acquisition part 11 may obtain the time range information indicating the selected arbitrary time range. [0092] in the input screen displayed on the display part 10 f, information for selecting or inputting the evaluation unit of the skill of the operator A (e.g., a pull-down menu, a check box, etc.) may be displayed. For example, in the input screen shown in FIG. 6, a pull-down menu for selecting a specific product (e.g., at least one of the products 1 to 3), a specific component (e.g., at least one of the components 1 to 7), or a specific operation (e.g., at least one of holding, storing, transporting, adjusting, inspecting, etc.) as the evaluation unit is displayed. The acquisition part 11 may acquire evaluation unit information indicating the evaluation unit that is selected or inputted. [0094] in FIG. 6, instead of the time range R0 associated with the product 1 from the timing T1 to the timing T2, a time range from a timing T1′ after the timing T1 to a timing T2′ after the timing T2 is selected as the arbitrary time range through the operation of the input part 10 e by the user. Further, the product 1 is selected as the evaluation unit.)
At the time the invention was filed, it would have been obvious for one of ordinary skill in the art to have modified the teachings of Kudo as taught by Wada, with a reasonable expectation of success if arriving at the claimed invention. One of ordinary skill in the art would have been motivated to make this modification to the teachings of Kudo with the motivation of, …provides a skill evaluation device, a skill evaluation method, and a storage medium capable of appropriately evaluating the skill of an operator by changing the time range for evaluating the skill of the operator.…determine the data of the specific time range used for calculating the index value, it is possible to improve the evaluation accuracy of the skill of the operator.…., as recited in Wada.
As per Claim 2, Kudo teaches: The work analyzing device as claimed in claim 1,
wherein the processor displays, as the work video image, an analysis result video on the video image indicator, wherein a rectangular frame surrounding each recognized … are overlaid on the analysis result video. (in at least [0053] At the playback time shown in FIG. 6A, since the work is normally performed as described above, the head position marker P 8 exists in the existence probability distribution P 9 and the hand position marker P 10 exists in the existence probability distribution P 11 It is present in. In this way, while comparing the Gantt chart P1 and the box beard figure P2 with the existence probability distributions P9 and P11, the user (administrator) can visually grasp what kind of action the worker performed it can. [0120] W, WA, WB work data management system 100, 100 A, 100 Ba, 100 Bb information processing apparatus 11 measurement interface 12 measurement data storage means 13 feature quantity extraction means 14 first input means 15 large process setting means 16 detailed process classification means 16 a event judgment Means 16 b Event determination list 16 c Process determination unit 16 d Process determination list 17 Detailed process storage unit 18 Work time totalization unit 19 Work time storage unit 20, 20 A Work time statistical analysis unit 21 Work operation summary unit 22 Work operation storage unit 23 Work operation statistics Analysis means 24, 24 A data visualization means 25 second input means 26 result display means (display means) 27 production efficiency analysis means 27 a step completion prediction means 27 b large step production plan 27 c step progress analysis means 27 d policy list (storage means) 27 e policy Selection means 3 1, 32, 33, 34 External interface 200 Data integration device 201 External interface 202 Data management means 203 Integrated data storage means 204 Identical product judgment list G 1 First measurement device G 2 Second measurement device N Network P 1 Gantt chart P 2 box beard drawing P 5 Playback time bar (line showing time at playback) P6 Working deviation marker P7 Flow line variation abnormal marker P9, P11 Probability probability distribution P12, P13 Flow line history P21 line (line indicating planning completion time of large process) P22 line Line indicating predicted completion time of large process) Pa 1 element (element of Gantt chart))
Although implied, Kudo does not expressly disclose the following limitations, which however, are taught by Wada,
…target object, and a text indicating the type of each recognized target object are overlaid… (in at least [0091] FIG. 6 is a view showing an example of an input screen displayed on the display part 10 f according to the present embodiment. In the input screen shown in FIG. 6, the operation status data 12 a for each predetermined part (herein, each of the right hand and the left hand) are displayed on a time axis. Through the operation of the input part 10 e by the user, an arbitrary time range is selected from the operation status data 12 a displayed on the time axis. When the time range selection button is pressed through the operation of the input part 10 e by the user, the acquisition part 11 may obtain the time range information indicating the selected arbitrary time range. [0092] in the input screen displayed on the display part 10 f, information for selecting or inputting the evaluation unit of the skill of the operator A (e.g., a pull-down menu, a check box, etc.) may be displayed. For example, in the input screen shown in FIG. 6, a pull-down menu for selecting a specific product (e.g., at least one of the products 1 to 3), a specific component (e.g., at least one of the components 1 to 7), or a specific operation (e.g., at least one of holding, storing, transporting, adjusting, inspecting, etc.) as the evaluation unit is displayed. The acquisition part 11 may acquire evaluation unit information indicating the evaluation unit that is selected or inputted. [0094] in FIG. 6, instead of the time range R0 associated with the product 1 from the timing T1 to the timing T2, a time range from a timing T1′ after the timing T1 to a timing T2′ after the timing T2 is selected as the arbitrary time range through the operation of the input part 10 e by the user. Further, the product 1 is selected as the evaluation unit.)
The reason and rationale to combine Kudo and Wada is the same as recited above.
As per Claim 3, Kudo teaches: The work analyzing device as claimed in claim 1,
wherein the processor identifies at least one of a component or a work tool as the type of the target objects. (in at least [0011] The work data management system W shown in FIG. 1 is a system for managing data (referred to as work data) relating to work at a work site. In the present embodiment, in order to manufacture a prescribed product, a state in which an operator repeatedly performs a predetermined operation (processing of a member, etc.) is photographed and analysis and display of work are performed based on the imaging result There. The above-mentioned member may be substantially the same as the finished product, or it may be a component of a finished product. [0032] For example, in the detailed process A1 shown in FIG. 4, the process determining unit 16c (see FIG. 2) performs the following process. That is, when the large step γ is "processing" and the event E 2 occurs within 10 seconds after the occurrence of the event E 1, and the state satisfying both of these continues for more than 1 second, the process determining means 16 c , It is determined that "the worker is setting a member in the processing machine". As described above, the detailed process classifying means 16 (see FIG. 2) classifies the work performed by the worker at the work site into a plurality of time-series detailed processes based on the work data including the image data of the work site. Further, the detailed process classifying means 16 specifies the detailed process performed by the operator based on the change in the position of the predetermined part (head and hand) on the body of the worker. [0110] In each of the embodiments, the example in which the work data management system W extracts the position of the head and hand of the worker as the feature amount has been described, but the present invention is not limited to this. For example, the position of a device or a tool (not shown) may be added. This makes it possible to further improve the determination accuracy of the detailed process. [0120] W, WA, WB work data management system 100, 100 A, 100 Ba, 100 Bb information processing apparatus 11 measurement interface 12 measurement data storage means 13 feature quantity extraction means 14 first input means 15 large process setting means 16 detailed process classification means 16 a event judgment Means 16 b Event determination list 16 c Process determination unit 16 d Process determination list 17 Detailed process storage unit 18 Work time totalization unit 19 Work time storage unit 20, 20 A Work time statistical analysis unit 21 Work operation summary unit 22 Work operation storage unit 23 Work operation statistics Analysis means 24, 24 A data visualization means 25 second input means 26 result display means (display means) 27 production efficiency analysis means 27 a step completion prediction means 27 b large step production plan 27 c step progress analysis means 27 d policy list (storage means) 27 e policy Selection means 3 1, 32, 33, 34 External interface 200 Data integration device 201 External interface 202 Data management means 203 Integrated data storage means 204 Identical product judgment list G 1 First measurement device G 2 Second measurement device N Network P 1 Gantt chart P 2 box beard drawing P 5 Playback time bar (line showing time at playback) P6 Working deviation marker P7 Flow line variation abnormal marker P9, P11 Probability probability distribution P12, P13 Flow line history P21 line (line indicating planning completion time of large process) P22 line Line indicating predicted completion time of large process) Pa 1 element (element of Gantt chart))
As per Claim 4, Kudo teaches: The work analyzing device as claimed in claim 3,
wherein the processor identifies at least one of an action of holding a component and an action of using a work tool as the type of the actions of the worker. (in at [0032] For example, in the detailed process A1 shown in FIG. 4, the process determining unit 16c (see FIG. 2) performs the following process. That is, when the large step γ is "processing" and the event E 2 occurs within 10 seconds after the occurrence of the event E 1, and the state satisfying both of these continues for more than 1 second, the process determining means 16 c , It is determined that "the worker is setting a member in the processing machine". As described above, the detailed process classifying means 16 (see FIG. 2) classifies the work performed by the worker at the work site into a plurality of time-series detailed processes based on the work data including the image data of the work site. Further, the detailed process classifying means 16 specifies the detailed process performed by the operator based on the change in the position of the predetermined part (head and hand) on the body of the worker. [0110] In each of the embodiments, the example in which the work data management system W extracts the position of the head and hand of the worker as the feature amount has been described, but the present invention is not limited to this. For example, the position of a device or a tool (not shown) may be added. This makes it possible to further improve the determination accuracy of the detailed process. [0120] W, WA, WB work data management system 100, 100 A, 100 Ba, 100 Bb information processing apparatus 11 measurement interface 12 measurement data storage means 13 feature quantity extraction means 14 first input means 15 large process setting means 16 detailed process classification means 16 a event judgment Means 16 b Event determination list 16 c Process determination unit 16 d Process determination list 17 Detailed process storage unit 18 Work time totalization unit 19 Work time storage unit 20, 20 A Work time statistical analysis unit 21 Work operation summary unit 22 Work operation storage unit 23 Work operation statistics Analysis means 24, 24 A data visualization means 25 second input means 26 result display means (display means) 27 production efficiency analysis means 27 a step completion prediction means 27 b large step production plan 27 c step progress analysis means 27 d policy list (storage means) 27 e policy Selection means 3 1, 32, 33, 34 External interface 200 Data integration device 201 External interface 202 Data management means 203 Integrated data storage means 204 Identical product judgment list G 1 First measurement device G 2 Second measurement device N Network P 1 Gantt chart P 2 box beard drawing P 5 Playback time bar (line showing time at playback) P6 Working deviation marker P7 Flow line variation abnormal marker P9, P11 Probability probability distribution P12, P13 Flow line history P21 line (line indicating planning completion time of large process) P22 line Line indicating predicted completion time of large process) Pa 1 element (element of Gantt chart))
As per Claim 5, Kudo teaches: The work analyzing device as claimed in claim 1,
wherein, when detecting an abnormality in the work performance status of the worker, the processor outputs the analysis result display information that includes information on the detected abnormality. (in at least [0049] The reproduction time bar P 5 (described as "Now" on the upper side) shown in FIG. 6A is a line indicating the reproduction time of the video displayed in the measurement data display areas P 3 and P 4. When images are being reproduced in the measurement data display areas P 3 and P 4, the reproduction time bar P 5 moves to the right side of the screen. In this manner, when the data visualizing means 24 displays the video data on the result display means 26 as a moving picture, it causes the Gantt chart P 1 to superimpose and display a line (reproduction time bar P 5) indicating the time at the time of reproduction of the video data. The work deviation marker P 6 and the flow line variation abnormality marker P 7 displayed on the Gantt chart P 1 will be described later. [0050] The measurement data display areas P 3 and P 4 are areas for displaying images of a work site at a predetermined reproduction time, a presence probability distribution of head / hand positions, and the like. In the example shown in FIG. 6A, in the Gantt chart P 1, the reproduction time bar P 5 exists in the detailed step A 1. In the row of the chart of the detailed step A1, the work deviation marker P6 and the flow line variation abnormality marker P7 are not displayed, and the width of the box beard figure P2 is narrow. In this way, when the detailed step A1 is normally repeated, the data visualizing means 24 displays the measurement result (photographed result) of the first measuring device G1 in the measurement data display area P3 and the data visualizing means 24 of the second measuring device G2 And displays the measurement result (imaging result) in the measurement data display area P 4.)
As per Claim 6, Kudo teaches: The work analyzing device as claimed in claim 5, wherein the analysis result display information includes, as the information on the detected abnormality, at least one of
a highlighted display of fields for the recognized target objects, a display of a mark indicating a corresponding one of the actions associated with the abnormality, a display of a text indicating the detected abnormality, and a display of the number of abnormality detection during a prescribed period of time. (in at least [0032] For example, in the detailed process A1 shown in FIG. 4, the process determining unit 16c (see FIG. 2) performs the following process. That is, when the large step γ is "processing" and the event E 2 occurs within 10 seconds after the occurrence of the event E 1, and the state satisfying both of these continues for more than 1 second, the process determining means 16 c , It is determined that "the worker is setting a member in the processing machine". As described above, the detailed process classifying means 16 (see FIG. 2) classifies the work performed by the worker at the work site into a plurality of time-series detailed processes based on the work data including the image data of the work site. Further, the detailed process classifying means 16 specifies the detailed process performed by the operator based on the change in the position of the predetermined part (head and hand) on the body of the worker. [0049] The reproduction time bar P 5 (described as "Now" on the upper side) shown in FIG. 6A is a line indicating the reproduction time of the video displayed in the measurement data display areas P 3 and P 4. When images are being reproduced in the measurement data display areas P 3 and P 4, the reproduction time bar P 5 moves to the right side of the screen. In this manner, when the data visualizing means 24 displays the video data on the result display means 26 as a moving picture, it causes the Gantt chart P 1 to superimpose and display a line (reproduction time bar P 5) indicating the time at the time of reproduction of the video data. The work deviation marker P 6 and the flow line variation abnormality marker P 7 displayed on the Gantt chart P 1 will be described later. [0050] The measurement data display areas P 3 and P 4 are areas for displaying images of a work site at a predetermined reproduction time, a presence probability distribution of head / hand positions, and the like. In the example shown in FIG. 6A, in the Gantt chart P 1, the reproduction time bar P 5 exists in the detailed step A 1. In the row of the chart of the detailed step A1, the work deviation marker P6 and the flow line variation abnormality marker P7 are not displayed, and the width of the box beard figure P2 is narrow. In this way, when the detailed step A1 is normally repeated, the data visualizing means 24 displays the measurement result (photographed result) of the first measuring device G1 in the measurement data display area P3 and the data visualizing means 24 of the second measuring device G2 And displays the measurement result (imaging result) in the measurement data display area P 4. [0056] FIG. 6C shows a screen image when the reproduction time bar P 5 is positioned in the detailed step A 2. In the example shown in FIG. 6C, a flow line variation abnormality marker P 7 is displayed in the row of the chart of the detailed process A 2. The flow line variation abnormality marker P 7 is a marker displayed in association with the element Pa 1 of the Gantt chart P 1 when the operation (ie, flow line history) of the work repeated in the past largely fluctuates. Whether or not to display the flow line variation abnormality marker P 7 is determined based on the variance value of the flow line history P 12. [0057] As described above, when the variance value of a plurality of flow line histories is equal to or larger than the predetermined threshold value in the predetermined detailed process, the data visualizing means 24 adds a flow line variation abnormality to the element Pa 1 of the Gantt chart P 1 corresponding to this detailed process The marker P 7 is displayed. [0058] In addition, when there is an abnormality in the flow line variation, the data visualizing means 24 displays the image of the side of the worker's head / hand where the variance value of the flow line history exceeds the predetermined threshold in the measurement data display areas P 3, P 4 As shown in FIG. In the example shown in FIG. 6C, since the variance value of the flow line history of the hand exceeds the predetermined threshold value, the position marker P 10 of the hand and the existence probability distribution P 11 of the hand are superimposed and displayed together with the image of the foreground in the measurement data display area P 3 There. In addition, in the measurement data display area P 4, the position marker P 10 of the hand and the flow line history P 13 of the hand are superimposed and displayed together with the image of the foreground. By displaying the flow line history P 13 in addition to the Gantt chart P 1 or the like, the user can efficiently confirm the information on the flow line variation abnormality.)
As per Claim 7 for a method (see at least Kudo [0019]), substantially recite the subject matter of Claim 1 and are rejected based on the same reasoning and rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PO HAN MAX LEE whose telephone number is (571)272-3821. The examiner can normally be reached on Mon-Thurs 8:00 am - 7:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rutao Wu can be reached on (571) 272-6045. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PO HAN LEE/Primary Examiner, Art Unit 3623
Prosecution Insights