Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendment
This is in response to applicant’s amendment/response filed on 09/18/2025, which has been entered and made of record. Claims 9, 11-32, 34-43 are pending in the application.
Response to Arguments
Applicant's arguments filed on 09/18/2025 regarding claims rejection under 35 U.S.C 103 have been fully considered but they are not persuasive.
Applicant submits “Linnell is seen to disclose to display a warning event with a specific color, e.g. the red in the specification, on a time series. In Figs. 8B and 8C of Linnell, the parameters Al to A6 are displayed intersecting each other in a single visual curve window 814, and the parameters of the operational components are not displayed in the different areas. Linnell, as admitted in the Office Action, fails to disclose or to suggest at least the features of, wherein a first warning event of the first robot and a second warning event of the second robot are specified based on the operation parameters.”
The examiner disagrees with Applicant’s premises and conclusion. In regard to Linnell, the prior art discloses one display window for one robot. However, Linnell has two or more robot as shown in Fig. 9A and 9B, it is obvious to have separated window or display screen for each robot. When there are separated display windows, the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first window and the second warning event is displayed with the second operation parameter in the second window. The claim would have been obvious because a particular known technique was recognized as part of the ordinary capabilities of one skilled in the art.
Applicant submits “That is, Lee searches the layout frame based on the Search term. Therefore the combination of Linnell and Lee does not disclose nor suggest the features of, wherein a first warning event of the first robot and a second warning event of the second robot are specified based on the operation parameters, and wherein a display in animation of a model of the first robot or the second robot is displayed in the operation display area, and a drawn frame in which the first warning event or the second warning event occurs is searched, wherein if a drawn frame in which the first warning event or the second warning event occurs is detected, the display in animation in the operation display area is updated to a state of the drawn frame that has been searched.”.
The examiner disagrees with Applicant’s premises and conclusion. Linnell teaches two or more robots as shown in Fig. 9A and 9B. Linnell also teaches “a first warning event of the first robot and a second warning event of the second robot are specified based on the operation parameters, and wherein a display in animation of a model of the first robot or the second robot is displayed in the operation display area” in Fig. 8A to Fig. 8D because when there are two or more robots, it is obvious that each robot has its own set of operation parameters and warning events. Linnell further detects warning event and displays warning even. What Linnell missing is searching the draw frame to detect a warning event. Lee teaches “When the search term for searching the layout frame is input, the electronic device 101 may search a frame corresponding to the search term in operation 1405”. One of ordinary skill in the art can see the improvement made in Lee for searching the warning events in Linnell.
Applicant submits “the Examiner asserts that Linnell and Lee are in similar fields because they both relate to "display interfaces", but the Applicant believes this is too superficial to meet the case law standard for "analogous art". The present invention provides a user interface that allows efficient verification of instantaneous warning events in a virtual robot simulation without overlooking them. Linnell is from an analogous art to that of the present claims. Meanwhile, the object of Lee is summarizing the content of video taken on consumer electronic devices to help users quickly understand the content. Linnell and Lee have completely different technical backgrounds, purposes, and target users. There is no incentive for an industrial robot simulation expert to apply a technology developed for the purpose of improving video thumbnail display to the completely different task of verifying robot operation. Therefore, Lee is prior art in a dissimilar field unrelated to Linnell and unrelated to the problem of the present invention. Besides, Linnell's technology focuses on real-time, closed-loop control, and the idea of searching for specific frames in pre-recorded simulation data defeats the purpose that system. Furthermore, Lee's "search" is directed at "objects" not at the logical occurrence of "warning events" as identified by Linnell. Therefore, the combination of the two technologies is technically unnatural and illogical, and would not be suggested to a person skilled in the art. Accordingly, the Applicant asserts that the proposed combination fails to meet the requirements to set forth a prima facie case of obviousness, and the rejections are traversed.”
The examiner disagrees with Applicant’s premises and conclusion. Both Linnell and Lee present contents on displays. Linnell has event data on screen that needs an operator to pay attention to. Lee teaches when a search term is inputted, the electronic device may extract a frame corresponding to the search term by searching the entirety or part of stored video data. Lee’s feature is exactly what the operator needs to search for event data. Therefore, when coding for a display software of robot simulation, it is obvious for one of ordinary skill in the art to incorporate the teaching of Lee to search the warning events.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Linnell et al. (US Pub 2016/0136815 A1) in view of Lee et al. (US Pub 2018/0033463 A1), DENSO (DENSO EMU multirobot offline software (Dual VS-A Demo), Youtube video, https://www.youtube.com/watch?v=ZK8hRUTvskY, Feb 6, 2014) and Parker, Lynne E. ("ALLIANCE: An architecture for fault tolerant multirobot cooperation." IEEE transactions on robotics and automation 14.2 (1998): 220-240.), further in view of Yajima et al. (US Pub 2015/0328774 A1).
As to claim 9, Linnell discloses an information processing apparatus configured to display information related to operation parameters of a robot system comprising a first robot and a second robot (Fig. 3B, first robot 314 and second robot 320, Fig. 9B, ¶0142, “control system 908 may receive a first data stream 918 from a first robotic device 910 operating within a workcell, a second data stream 920 from a second robotic device 912 operating within the workcell” ¶0080, “The interface may allow a user to include one or more nodes 504 which may represent components of the building process, such as robot nodes representing different types and/or configurations of robots” ¶0089, “the display window 510 may provide users with multiple 3D views of the physical workcell” ¶0119, “FIG. 8B shows another point along timeline 810 that occurs at a later point during a programmed sequence of operations for the robotic device 804. In FIG. 8B, display window 802 illustrates the movement of robotic device 804 along motion path 806.”),
wherein a first warning event of the first robot and a second warning event of the second robot are specified based on the operation parameters (Fig. 8B, ¶0121, “the visual simulation may contain warning signals indicating possible errors or conflicts resulting from robotic operation.” ¶0122, “three possible errors 820 (reach, flip, singularity) are shown within visual curve window 814.” ¶0126, “a user interface may include a display window showing parameters of robotic operation at particular steps or timestamps of a robotic process.”), and
wherein, on a display unit,
a first area displaying a first operation parameter serving as an operation parameter corresponding to the first robot in a time series (Fig. 8B) and a second area displaying a second operation parameter serving as an operation parameter corresponding to the second robot in a time series are displayed (Fig.8B, Fig. 8D, Fig.9B, ¶0127, “joint parameters 854 and 856 may represent current joint parameters (e.g., joint angles, velocities, or accelerations) of different robotic devices at the current trajectory step 852.” Bot_a and Bot_b are two different robots. Since Linnell has two robots, it is obvious to have two windows or display screens for each robot.),
an operation display area displaying an operation of the first robot or the second robot operating simulatively in a virtual environment is displayed (Linnell, Fig.9E) and wherein a display in animation of a model of the first robot or the second robot is displayed in the operation display area (Linnell, Fig.9A to 9E).
Linnell does not explicitly disclose “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area.” and “a drawn frame in which the first warning event or the second warning event occurs is searched, wherein if a drawn frame in which the first warning event or the second warning event occurs is detected, the display in animation in the operation display area is updated to a state of the drawn frame that has been searched.”.
However, it is obvious to display information of different robots in different areas. Since Linnell discloses two robots in Fig.8D and Fig. 9A-9B, it is obvious to have separated window or display screen for each robot. When there are separated display windows, the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area. The claim would have been obvious because a particular known technique was recognized as part of the ordinary capabilities of one skilled in the art.
Lee teaches a drawn frame in which the first warning event or the second warning event occurs is searched, wherein if a drawn frame in which the first warning event or the second warning event occurs is detected, the display in animation in the operation display area is updated to a state of the drawn frame that has been searched (Lee, ¶0166, “When the search term for searching the layout frame is input, the electronic device 101 may search a frame corresponding to the search term in operation 1405. For example, the electronic device 101 may extract at least one frame including an object corresponding to the search term. For example, when a search term is inputted (1500) as illustrated in FIG. 15, the electronic device 101 may extract a frame corresponding to the search term by searching the entirety or part of stored video data.” ¶0167, “In operation 1407, the electronic device 101 may output a list of searched frames. For example, as illustrated in FIG. 15, the electronic device 101 may output the searched frames as a single search result 1510 without distinguishing the data. In another example, the electronic device 101 may output the searched frames as a search result by classifying them by video data (1520).”).
Linnell and Lee are considered to be analogous art because all pertain to display interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “a drawn frame in which the first warning event or the second warning event occurs is searched, wherein if a drawn frame in which the first warning event or the second warning event occurs is detected, the display in animation in the operation display area is updated to a state of the drawn frame that has been searched.” as taught by Lee. The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
DENSO teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area (DENSO shows two robot arms as controller 1 and controller 2. The operation parameter areas and error warning areas are separated for each controller.).
PNG
media_image1.png
1397
1980
media_image1.png
Greyscale
PNG
media_image2.png
264
1069
media_image2.png
Greyscale
Linnell and DENSO are considered to be analogous art because all pertain to robot simulation interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area.” as taught by DENSO. The suggestion/motivation would have been in order to simulate multi robots in a single work cell (DENSO).
Parker teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area. (Parker, Page 13-14, “we will refer to these robots individually as GREEN,BLUE, and GOLD.”
PNG
media_image3.png
885
1107
media_image3.png
Greyscale
Page 16,
PNG
media_image4.png
690
1146
media_image4.png
Greyscale
Linnell, DENSO and Parker are considered to be analogous art because all pertain to robot control interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area.” as taught by Parker. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
Yajima also teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area (Yajima, Fig. 7, ¶0112, “The allowable load b of the bearings and the allowable load g of the reduction gears are displayed, and the evaluation results of the broken conditions of the bearings and the reduction gears in the J1 joint to the J6 joint are displayed visually.”. Each joint could be interpreted as one robot or it is obvious to display a robot condition instead of one joint.)
PNG
media_image5.png
819
756
media_image5.png
Greyscale
)
Linnell, DENSO, Parker and Yajima are considered to be analogous art because all pertain to robot control interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area.” as taught by Yajima. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
As to claim 38, the combination of Linnell and Lee disclose an information processing method comprising: displaying information related to operation parameters of a robot system including a first robot and a second robot, wherein a first warning event of the first robot and a second warning event of the second robot are specified based on the operation parameters, and wherein, on a display unit, a first area displaying a first operation parameter serving as an operation parameter corresponding to the first robot in a time series and a second area displaying a second operation parameter serving as an operation parameter corresponding to the second robot in a time series are displayed, the first area and the second area are displayed separately, the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, and an operation display area displaying an operation of the first robot or the second robot operating simulatively in a virtual environment is displayed, and wherein a display in animation of a model of the first robot or the second robot is displayed in the operation display area, and a drawn frame in which the first warning event or the second warning event occurs is searched, wherein if a drawn frame in which the first warning event or the second warning event occurs is detected, the display in animation in the operation display area is updated to a state of the drawn frame that has been searched (See claim 9 for detailed analysis.).
Claims 11-18, 27-31, 34-37, 39-42 are rejected under 35 U.S.C. 103 as being unpatentable over Linnell et al. (US Pub 2016/0136815 A1) and Lee et al. (US Pub 2018/0033463 A1) further in view of DENSO (DENSO EMU multirobot offline software (Dual VS-A Demo), Youtube video, https://www.youtube.com/watch?v=ZK8hRUTvskY, Feb 6, 2014) and Parker, Lynne E. ("ALLIANCE: An architecture for fault tolerant multirobot cooperation." IEEE transactions on robotics and automation 14.2 (1998): 220-240.), Yajima et al. (US Pub 2015/0328774 A1).
As to claim 11, claim 9 is incorporated and Linnell discloses
a first area displaying a first operation parameter serving as an operation parameter corresponding to the first robot in a time series (Fig. 8B) and a second area displaying a second operation parameter serving as an operation parameter corresponding to the second robot in a time series are displayed (Fig.8B, Fig. 8D, Fig.9B, ¶0127, “joint parameters 854 and 856 may represent current joint parameters (e.g., joint angles, velocities, or accelerations) of different robotic devices at the current trajectory step 852.” Bot_a and Bot_b are two different robots. Since Linnell has two robots, it is obvious to have two windows or display screens for each robot.), and
Linnell does not explicitly disclose the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area.
However, it is obvious to display information of different robots in different areas. Since Linnell discloses two robots in Fig.8D and Fig. 9A-9B, it is obvious to have separated window or display screen for each robot. When there are separated display windows, the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area. The claim would have been obvious because a particular known technique was recognized as part of the ordinary capabilities of one skilled in the art.
DENSO teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area (DENSO shows two robot arms as controller 1 and controller 2. The operation parameter areas and error warning areas are separated for each controller.).
PNG
media_image1.png
1397
1980
media_image1.png
Greyscale
PNG
media_image2.png
264
1069
media_image2.png
Greyscale
Linnell and DENSO are considered to be analogous art because all pertain to robot simulation interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area.” as taught by DENSO. The suggestion/motivation would have been in order to simulate multi robots in a single work cell (DENSO).
Parker teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area. (Parker, Page 13-14, “we will refer to these robots individually as GREEN,BLUE, and GOLD.”
PNG
media_image3.png
885
1107
media_image3.png
Greyscale
Page 16,
PNG
media_image4.png
690
1146
media_image4.png
Greyscale
Linnell, DENSO and Parker are considered to be analogous art because all pertain to robot control interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area.” as taught by Parker. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
Yajima also teaches the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area (Yajima, Fig. 7, ¶0112, “The allowable load b of the bearings and the allowable load g of the reduction gears are displayed, and the evaluation results of the broken conditions of the bearings and the reduction gears in the J1 joint to the J6 joint are displayed visually.”. Each joint could be interpreted as one robot or it is obvious to display a robot condition instead of one joint.)
PNG
media_image5.png
819
756
media_image5.png
Greyscale
)
Linnell, DENSO, Parker and Yajima are considered to be analogous art because all pertain to robot control interface. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “the first area and the second area are displayed separately, and the first warning event is displayed with the first operation parameter in the first area and the second warning event is displayed with the second operation parameter in the second area, the first warning event and the second warning event are respectively independently displayed in the first area and in the second area.” as taught by Yajima. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
As to claim 12, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first area and the second area are displayed in a same area in a separated state (DENSO, It shows property of controller 1 and controller 2 by user selection.
PNG
media_image6.png
1355
1966
media_image6.png
Greyscale
PNG
media_image7.png
845
430
media_image7.png
Greyscale
)
As to claim 13, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose a state in which the first area is displayed and a state in which the second area is displayed can be switched (DENSO, The property display can be switched by user selecting controller 1 and controller 2.
PNG
media_image7.png
845
430
media_image7.png
Greyscale
).
As to claim 14, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose a state in which the first area is displayed and a state in which the second area is displayed are switched by selecting the first robot or the second robot (DENSO, The property display can be switched by user selecting controller 1 and controller 2.
PNG
media_image7.png
845
430
media_image7.png
Greyscale
).
As to claim 15, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first warning event is displayed in a manner superimposed on the first operation parameter, and the second warning event is displayed in a manner superimposed on the second operation parameter (Linnell, Fig.8c.
PNG
media_image8.png
297
300
media_image8.png
Greyscale
).
As to claim 16, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose in displaying the first warning event and the second warning event, display is realized with a different display format depending on a stage of the first warning event and the second warning event (Linnell, Fig.8c. ¶0123, “the warning signal may be represented by changing the color of the robotic device 804 within the visual simulation to a different color (e.g., red). In some examples, the color of the timeline 810 could be changed as well or instead.” ¶0124, “the warning signal may be indicated within one or more visual curve representations as well or instead. For instance, visual curve window 814 may indicate a warning signal as a colored or shaded section 818, and may further indicate the type of error or warning by indicating a selection from a list of possible errors 820. For example, reach may be highlighted, changed to red, or otherwise selected to indicate an error in that robot actor 804 cannot reach a particular programmed target location. In additional examples, a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device. In further examples, separate warning signals may be shown both within a geometric representation of robotic operation and within a HUD or other type of visual representation as well.”).
As to claim 17, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose in displaying the first warning event and the second warning event, display is realized with a different color depending on a stage of the first warning event and the second warning event (Linnell, Fig.8c. ¶0123, “the warning signal may be represented by changing the color of the robotic device 804 within the visual simulation to a different color (e.g., red). In some examples, the color of the timeline 810 could be changed as well or instead.” ¶0124, “the warning signal may be indicated within one or more visual curve representations as well or instead. For instance, visual curve window 814 may indicate a warning signal as a colored or shaded section 818, and may further indicate the type of error or warning by indicating a selection from a list of possible errors 820. For example, reach may be highlighted, changed to red, or otherwise selected to indicate an error in that robot actor 804 cannot reach a particular programmed target location. In additional examples, a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device. In further examples, separate warning signals may be shown both within a geometric representation of robotic operation and within a HUD or other type of visual representation as well.”).
As to claim 18, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose in displaying the first warning event and the second warning event, display is realized with a different display pattern depending on a stage of the first warning event and the second warning event (Linnell, Fig. 8B-8C, ¶0124, “visual curve window 814 may indicate a warning signal as a colored or shaded section 818, and may further indicate the type of error or warning by indicating a selection from a list of possible errors 820. For example, reach may be highlighted, changed to red, or otherwise selected to indicate an error in that robot actor 804 cannot reach a particular programmed target location. In additional examples, a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device. In further examples, separate warning signals may be shown both within a geometric representation of robotic operation and within a HUD or other type of visual representation as well.”).
As to claim 27, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first warning event or the second warning event is updated to a state displayed in the first area or the second area based on a predetermined operation of a user (Linnell, ¶0124, “a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device.”).
As to claim 28, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose based on a predetermined operation of a user, a state in which a state of the first robot or the second robot at a predetermined timing on the time series is displayed in the first area or the second area is updated to a state in which the first warning event or the second warning event at a timing closest to the predetermined timing anterior or posterior thereto on the time series is displayed in the first area or the second area (Linnell, ¶0124, “a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device.” ¶0133, “a user of computing device 902 may specify that control system 908 should watch joint parameters of a particular robotic device to ensure that the robotic device doesn't violate a constraint of motion. The user may also specify how the control system 908 should respond when the robot approaches or violates the constraint. For example, control system 908 may be given instruction to slow down the robot or stop the robot in such a circumstance.”).
As to claim 29, claim 28 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein, in a state where a state of the first robot or the second robot at the predetermined timing is displayed in the first area or the second area, when the user performs an operation to display the warning event that exists anterior to the predetermined timing on the time series, update is performed to a state in which the first warning event or the second warning event at a timing closest and anterior to the predetermined timing on the time series is displayed in the first area or the second area, and, wherein, in a state where a state of the first robot or the second robot at the predetermined timing is displayed in the first area or the second area, when the user performs an operation to display the first warning event or the second warning event that exists posterior to the predetermined timing on the time series, update is performed to a state in which the first warning event or the second warning event at a timing closest and posterior to the predetermined timing on the time series is displayed the first area or the second area (Linnell, ¶0134, “a user of computing device 902 may specify that control system 908 should watch the motion path of a particular robotic device to ensure that it doesn't deviate from a programmed motion path (e.g., programmed from a graphical interface of computing device 902). If the motion path does deviate by more than a threshold amount, instructions may be provided for how control system 908 should respond. For example, control system 908 may be instructed to adjust the motion path of the robotic device on the fly to coincide with the pre-programmed motion path, or to control the robotic device to hold position until a human user can intervene.”).
As to claim 30, claim 27 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein a plurality of the first warning events or the second warning events are specified on the time series, and based on the predetermined operation, update is performed from a state in which a predetermined warning event among the plurality of first warning events or the second warning events is displayed in the first area or the second area to a state in which a warning event at a timing closest to the predetermined warning event anterior or posterior thereto on the time series is displayed in the first area or the second area (Linnell, ¶0124, “reach may be highlighted, changed to red, or otherwise selected to indicate an error in that robot actor 804 cannot reach a particular programmed target location. In additional examples, a curve or numerical parameter may change color (e.g., to red) when it approaches or reaches a predefined constraint or user-defined constraint for the robotic device.” ¶0133-0134).
As to claim 31, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein the operation display area and the first area and the second area are displayed so as to be linked (DENSO,
PNG
media_image9.png
1212
2001
media_image9.png
Greyscale
).
As to claim 34, claim 9 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein a type of the first warning event or the second warning event being specified and a threshold value specifying the first warning event or the second warning event can be set by a user (Linnell, ¶0125, “a warning signal may be triggered when the motion path taken by a robotic device within the workcell deviates from a pre-programmed motion path for the robotic device (e.g., by more than a threshold amount or error tolerance).” ¶0134, “a user of computing device 902 may specify that control system 908 should watch the motion path of a particular robotic device to ensure that it doesn't deviate from a programmed motion path (e.g., programmed from a graphical interface of computing device 902). If the motion path does deviate by more than a threshold amount, instructions may be provided for how control system 908 should respond.”).
As to claim 35, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein a first movable range being a movable range of a joint of the first robot is displayed in the first area, and a second movable range being a movable range of a joint of the second robot is displayed in the second area (Linnell, ¶0125, “a warning signal may be triggered when the motion path taken by a robotic device within the workcell deviates from a pre-programmed motion path for the robotic device (e.g., by more than a threshold amount or error tolerance).” ¶0134, “a user of computing device 902 may specify that control system 908 should watch the motion path of a particular robotic device to ensure that it doesn't deviate from a programmed motion path (e.g., programmed from a graphical interface of computing device 902). If the motion path does deviate by more than a threshold amount, instructions may be provided for how control system 908 should respond.”).
As to claim 36, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose a time base is displayed in the first area or the second area (Linnell, Fig. 8A-8C).
As to claim 37, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first robot and the second robot differ (Linnell, Fig. 3B, 9E).
As to claim 39, claim 9 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose a manufacturing method for manufacturing articles by causing the first robot or the second robot to execute an operation verified by using the information processing apparatus according to claim 9. (See claim 9 for detailed analysis.).
As to claim 40, claim 38 is incorporated and the combination of Linnell, Lee, DENSO and Parker disclose a non-transitory computer-readable medium storing a program configured to have a computer execute the image processing method according to claim 38. (See claim 9 for detailed analysis.).
As to claim 41, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first area and the second area are displayed in the same window of the display unit in such a manner that the first area and the second area are separated (Yajima, Fig. 7, ¶0112, “The allowable load b of the bearings and the allowable load g of the reduction gears are displayed, and the evaluation results of the broken conditions of the bearings and the reduction gears in the J1 joint to the J6 joint are displayed visually.”. Each joint could be interpreted as one robot or it is obvious to display a robot condition instead of one joint.)
PNG
media_image5.png
819
756
media_image5.png
Greyscale
).
As to claim 42, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose the first warning event is displayed so as to superimpose on the first operation parameter in such a manner that an user can visually recognize the first operation parameter and the second warning event is displayed so as to superimpose on the second operation parameter in such a manner that an user can visually recognize the second operation parameter (Yajima, Fig. 7, ¶0112, “The allowable load b of the bearings and the allowable load g of the reduction gears are displayed, and the evaluation results of the broken conditions of the bearings and the reduction gears in the J1 joint to the J6 joint are displayed visually.”. Each joint could be interpreted as one robot or it is obvious to display a robot condition instead of one joint.)
PNG
media_image5.png
819
756
media_image5.png
Greyscale
).
As to claim 43, claim 11 is incorporated and the combination of Linnell, Lee, DENSO, Parker and Yajima disclose wherein, the first area and the second area are displayed separately (see claim 11 for details.).
Claims 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Linnell et al. (US US Pub 2016/0136815 A1) in view of Lee et al. (US Pub 2018/0033463 A1), DENSO (DENSO EMU multirobot offline software (Dual VS-A Demo), Youtube video, https://www.youtube.com/watch?v=ZK8hRUTvskY, Feb 6, 2014) and Parker, Lynne E. ("ALLIANCE: An architecture for fault tolerant multirobot cooperation." IEEE transactions on robotics and automation 14.2 (1998): 220-240.), further in view of Yajima et al. (US Pub 2015/0328774 A1) and Sato et al. (US Pub 2012/0182155 A1).
As to claim 19, claim 11 is incorporated and the combination of Linnell, Lee, DENSO and Parker do not explicitly disclose wherein, in displaying the first warning event and the second warning event, display is realized with a different density depending on a stage of the first warning event and the second warning event.
Sato discloses in displaying the first warning event and the second warning event, display is realized with a different density depending on a stage of the first warning event and the second warning event (Sato, ¶0042, “determine the display format so that a brightness when the reaching time is a first time is higher than a brightness when the reaching time is a second time which is longer than the first time, and to generate the image data for illustrating the position/posture in the determined display format.” ¶0282, “when the time until the robot 1200 reaches the display position is short, the image generation unit 1104 sets the brightness of the image 2902 of the robot 1200 to a high value. That is to say, the image generation unit 1104 causes the image 2902 to be displayed brightly. On the other hand, when the time until the robot 1200 reaches the display position is long, the image generation unit 1104 sets the brightness of the image 2902 of the robot 1200 to a low value.” Brightness is the density of light. Brighter light has higher density.).
Linnell, DENSO, Parker, and Sato are considered to be analogous art because all pertain to robot system. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “in displaying the first warning event and the second warning event, display is realized with a different density depending on a stage of the first warning event and the second warning event.” as taught by Sato. The suggestion/motivation would have been in order to make the image more noticeable (Sato, ¶0283).
As to claim 20, claim 11 is incorporated and the combination of Linnell, Lee, DENSO and Parker do not explicitly disclose wherein, in displaying the first warning event and the second warning event, display is realized with a different brightness depending on a stage of the first warning event and the second warning event.
Sato discloses wherein, in displaying the first warning event and the second warning event, display is realized with a different brightness depending on a stage of the first warning event and the second warning event (Sato, ¶0042, “determine the display format so that a brightness when the reaching time is a first time is higher than a brightness when the reaching time is a second time which is longer than the first time, and to generate the image data for illustrating the position/posture in the determined display format.” ¶0282, “when the time until the robot 1200 reaches the display position is short, the image generation unit 1104 sets the brightness of the image 2902 of the robot 1200 to a high value. That is to say, the image generation unit 1104 causes the image 2902 to be displayed brightly. On the other hand, when the time until the robot 1200 reaches the display position is long, the image generation unit 1104 sets the brightness of the image 2902 of the robot 1200 to a low value.” Brightness is the density of light. Brighter light has higher density.).
Linnell, DENSO, Parker, and Sato are considered to be analogous art because all pertain to robot system. It would have been obvious before the effective filing date of the claimed invention to have modified Linnell with the features of “in displaying the first warning event and the second warning event, display is realized with a different brightness depending on a stage of the first warning event and the second warning event.” as taught by Sato. The suggestion/motivation would have been in order to make the image more noticeable (Sato, ¶0283).
Claims 21-26 are rejected under 35 U.S.C. 103 as being unpatentable over Linnell et al. (US US Pub 2016/0136815 A1) in view of Lee et al. (US Pub 2018/0033463 A1), DENSO (DENSO EMU multirobot offline software (Dual VS-A Demo), Youtube video, https://www.youtube.com/watch?v=ZK8hRUTvskY, Feb 6, 2014) and Parker, Lynne E. ("ALLIANCE: An architecture for fault tolerant multirobot cooperation." IEEE transactions on robotics and automation 14.2 (1998): 220-240.), further in view of Yajima et al. (US Pub 2015/0328774 A1) and Nakazato (US Pub 2016/0288318 A1).
As to claim 21, claim 11 is incorporated and the combination of Linnell, Lee, DENSO and Parker do not disclose in displaying the first warning event and the second warning event, a first stage of the first warning event and the second warning event is displayed with a first display format, and a second stage of the first warning event and the second warning event is displayed with a second display format that differs from the first display format.
Nakazato teaches in displaying the first warning event and the second warning event, a first stage of the first warning event and the second warning event is displayed with a first display format, and a second stage of the first warning event and the second warning event is displayed with a second display format that differs from the first display format (Nakazato, ¶0023, “the observer 1001 can check a possible interference between a robot arm and an obstacle. The possibility of damaging the robot when a robot arm is brought into contact with another robot arm in checking the interference can be reduced.“ ¶0096, “the interference determination unit 308 determines whether there is a portion where a robot arm is brought into contact with another robot arm and detects a possible interference portion.” ¶0096, “The interference determination unit 308 performs a robot simulation at a replay position advanced by the replay speed set by the setting unit 301 from the position and orientation of the robot at the one-loop previous replay position. Then, if there is a portion where the distance between the robot arm and a workpiece or a working space is equal to or less than a threshold value having been determined beforehand, the interference determination unit 308 determines that an interference may occur at the identified portion and detects it as a possible interference portion.” ¶0097, “the superimposed image generation unit 304 generates an image including an operating state of the second robot arm 1004 to be virtually displayed based on the position and orientation of the viewpoint acquired in step S3040, the three-dimensional shape information and the operation data acquired in step S3010, and the parameters set in step S3020. Further, the superimposed image generation unit 304 generates an image by emphasizing the possible contact portion detected in step S3047. More specifically, the superimposed image generation unit 304 displays a red-colored three-dimensional shape model of the robot in the vicinity of the detected portion.” ¶0100, “The interference determination unit 308 can be configured to detect a portion where the distance from the se