Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments filed 12/08/2025 have been fully considered but they are not persuasive. Claims 1-5,7-11, 13-17 have been amended. Claims 19-20 have been added. Applicant’s amendments have necessitated the new ground of rejection set forth herein; according, this action is made final.
Title:
Applicant has amended title. The object of title has been withdrawn.
Claim Rejections - 35 USC §112(b)
Applicant amended claims 1-5,7-11, 13-17 to overcome the rejection under 35 USC §112(b). The previous rejection of claims 1-18 under 35 USC §112(b) has been withdrawn.
Claim Rejections - 35 USC §102(a)/(1)
Applicant’s arguments with respect to independent claims have been considered but are moot because the rejection has been modified to address the newly added limitations. The Examiner now relies on the references OGAWA and Lee for the argued limitation.
Claim Objections
Claim 18 is objected to because of the following informalities: Claim 18 states Currently Amended but claim has not amended. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1-2, 7-8, 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over AKITA RYOHEI, JP2013084181-English translated, (“RYOHEI”) in view of OGAWA, U.S Patent Application Publication No.20230154016 (“OGAWA”) further in view of Lee, U.S Patent Application Publication No.20200012894 (“Lee”)
Regarding independent claim 1, RYOHEI teaches an information processing method implemented in a computer ([0001] The present invention relates to an image generating device, an image generating program, and an image display system having an image generating device.”), comprising:
obtaining a two-dimensional simulation image that is formed when a plurality of target subjects present in a three-dimensional simulation space is captured by a virtual camera (see at least [0002] “Mobile terminals, in-vehicle display devices, etc. are equipped with image generation devices that generate images for 2D display by projecting images of objects (subjects) placed in a 3D virtual space onto a 2D surface from a predetermined viewpoint.” [0007] “FIG. 1 is an example of a diagram showing the arrangement of objects in the three-dimensional virtual space having the X, Y, and Z axes described above. In FIG. 1, the horizontal axis indicates the distance in the Z-axis direction from a predetermined viewpoint position VP. The Z value of the viewpoint position VP is 0. Furthermore, objects m1 to m4 are arranged in order in the Z-axis direction with a predetermined virtual camera position (viewpoint position VP) as the reference. Here, as an example, it is assumed that objects m1 and m4 are opaque objects, and objects m2 and m3 are semi-transparent objects”; [0015] “A first aspect of an image generating device is an image generating device that generates, based on the transparency of each object in a three-dimensional virtual space and a depth value from a predetermined viewpoint position, an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position, the image generating device having: a first Z-buffer that stores the depth value of a semi-transparent first object; a second Z-buffer that stores the depth value of an opaque second object; and an image generating unit that calculates the blurriness of the first and second objects based on the depth values of the first and second objects and generates an image for two-dimensional display in which the first and second objects are blurred according to the blurriness of the first and second objects, wherein, when the first object is placed closer to the predetermined viewpoint position than the second object in the three-dimensional virtual space and the first object and the second object overlap in the image for two-dimensional display, the image generating unit corrects the blurriness of the area where the first object and the second object overlap by a proportion of the blurriness of the second object according to the transparency of the first object.”); and
generating a second simulation image that visually displays information indicating a degree of overlapping of the plurality of target subjects in the two-dimensional simulation image (see at least [0009] Figure 2 shows an example of an image in which the object in Figure 1 is projected onto a two-dimensional surface. The image in FIG. 2 shows an image in FIG. 1 having objects m1 to m4 that can be seen from a given viewpoint position VP. The symbol BG indicates the background portion. The background portion BG is an opaque object. In Figure 2, no blurring has been applied to any of the objects yet. [0010] Consider a case in which, in a three-dimensional virtual space, a semi-transparent object is placed closer to a predetermined viewpoint position VP than an opaque object, and the semi-transparent object and the opaque object overlap with each other based on this viewpoint position VP. Hereinafter, this state of object placement will be referred to as an opaque object being placed behind a semi-transparent object.”; [0015] “A first aspect of an image generating device is an image generating device that generates, based on the transparency of each object in a three-dimensional virtual space and a depth value from a predetermined viewpoint position, an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position, the image generating device having: a first Z-buffer that stores the depth value of a semi-transparent first object; a second Z-buffer that stores the depth value of an opaque second object; and an image generating unit that calculates the blurriness of the first and second objects based on the depth values of the first and second objects and generates an image for two-dimensional display in which the first and second objects are blurred according to the blurriness of the first and second objects, wherein, when the first object is placed closer to the predetermined viewpoint position than the second object in the three-dimensional virtual space and the first object and the second object overlap in the image for two-dimensional display, the image generating unit corrects the blurriness of the area where the first object and the second object overlap by a proportion of the blurriness of the second object according to the transparency of the first object.” [0016] According to the first aspect of the image generating device, the area where a translucent object and an opaque object overlap can be appropriately blurred according to the transparency of the translucent object, thereby generating a more natural image for two-dimensional display”) RYOHEI is understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, OGAWA teaches generating, for use as training data for a machine learning model, a second simulation image that visually displays information indicating a degree of overlapping of the plurality of target subjects, indicating the degree of overlapping (see at least [0098] In S1002, the blocking determination unit 207 determines the presence or absence of a blocking region where the candidate object is blocked based on the position of the candidate object in the current processing target image (second image). More specifically, the blocking determination unit 207 performs the blocking determination for each candidate object in the current image. The blocking determination processing in S1002 will be described in more detail with reference to FIG. 10B. In this case, the blocking determination unit 207 performs the blocking determination on a candidate of which no correlated candidate is found in S702 (referred to as an object of interest). First, in S10021, the blocking determination unit 207 determines whether the correlation is established for all candidate objects detected in the past in S702. When the correlation with candidate objects detected in the current image is completed for all of the candidate objects detected in the past image (first image), the processing proceeds to S10025. Among the candidate objects detected in the past image, if there is a past candidate object (object of interest) that has a similarity to a candidate object detected in the current image smaller than or equal to the threshold value, the processing proceeds to S10022. More specifically, when the processing proceeds to S10022, there may be a blocked candidate object. In S10022, for the current candidate object (object of interest), the blocking determination unit 207 acquires information indicating a degree of overlapping between the BB of the relevant candidate and the BB of another candidate. As an index indicating the degree of overlapping between objects, Intersection of Union (hereinafter referred to as IoU) is calculated. More specifically, when partial images (BBs) for the candidate objects detected in the current image are a region A of an object A and a region B of an object B, IoU for the object A and the object B is calculated as a region (A ∩ B)/(A U B). Higher IoU indicates a higher degree of overlapping between the objects. The other candidate object of which IoU exceeds a threshold value is set as an occluder of the relevant candidate. In this case, the status of the candidate object A is determined to be “Blocked”. In S10024, the position of a candidate determined to be “Blocked” by the blocking determination unit 207 is updated based on the position of the occluder. For example, an update may be made as Formula (2-1).[ 0101] A modification 2-1 performs the blocking determination by a neural network. An example of the blocking determination by the neural network is “Zhou, Bi-box Regression for Pedestrian Detection and Occlusion, In: ECCV2018”. In this example, in S1002, the tracking unit 205 estimates the BB of an object and simultaneously estimates a non-blocked region (viewable region) of an object region. Then, when the ratio of the region where blocking has occurred to the object region exceeds a predetermined threshold value, the blocking determination unit 207 can determine the blocking”; [0103] FIG. 11 illustrates images 1211, 1212, 1213, and 1214 acquired at the times t = 0, 1, 2, and 3, respectively, and a tracking target 1216. At the time t = 0, the tracking target 1216 and a similar object 1215 exist, and the two objects are in a state of being tracked. At the time t = 1, the tracking target 1216 is hidden by a similar object 1217, and only the similar object 1217 exists as a candidate object at the time t = 1. At this time, when IoU of 1216 and 1217 exceeds a threshold value and is determined to be blocking, the position of 1216 is updated to match the position of 1217 based on Formula (2-1). At the time t = 2, since the blocking is not resolved, the position of 1216 is updated to match the position of 1218, which is the occluder. At the time t = 3, the blocking is resolved, and three different candidates 1219, 1220, and 1221 exist. At this time, a correct correlation result is the correlations between 1218 and 1219 and between 1216 and 1220. However, if the position of 1216 is not updated to match the positions of 1217 and 1218 without the blocking determination, 1216 will exist around the candidate 1221 at the time t = 3. Thus, 1216 is highly likely to be correlated with the newly acquired candidate 1221, not 1220, possibly resulting in erroneous tracking. On the other hand, when the position of the candidate 1216 is updated based on Formula (2-1), the position of 1216 becomes close to the position of 1220, making it possible to correlate 1216 with 1220. Thus, the possibility of erroneous tracking can be reduced.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI with calculating degree overlapping between objects using a neural network as seen in OGAWA because this modification would determine a blocked candidate object ([0098] of OGAWA) Both RYOHEI and OGAWA are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Lee teaches generating, for use as training data for a machine learning model, a second image that visually displays information indicating a degree of overlapping of the plurality of target subjects by adding a numerical label to each of the plurality of target subjects , the numerical label indicating the degree (see at least Figs 11-14; [0046] FIG. 2 is a block diagram of the data flow of an active learning system for training a neural network, according some embodiments. For example, an initial setting of the active learning system 200 includes a neural network 210 initialized with random parameters, an initial set of labelled training images 201, a trainer 202, a set of unlabeled images 203. In this case, the neural network 210 is a user defined neural network. [0080] FIG. 11 shows a schematic of principles of evaluating the impact of bounding box geometry location according to one embodiment when multiple bounding boxes are detected together in a single image. In such a case, it is highly possible that the bounding boxes can overlap each other. For example, in image 1110, five bounding boxes 1101, 1101, 1103, 1004 and 1105, are detected, while four of these boxes, i.e., the bounding boxes 1101-1104 overlap. As a result, two of the bounding boxes (1101 and 1102) can only detect part of the occluded objects. [0081] In contrast, in image 1120, four bounding boxes 1121, 1122, 1123, 1124 and are detected, where three bounding boxes 1121-1123 overlap. By comparing images 1110 and 1120, the image 1110 is more challenging and require more attention by humans to examine. [0083] FIG. 12 shows a schematic of principles of evaluating diversity and classification metrics jointly according to some embodiments. FIG. 12 shows two images (1201 and 1202) with overlapped bounding boxes having different classification metric. In image 1201, two bounding boxes 1203 and 1204 actually detect the same object, which is a partially-visible car. Since only part of the car is visible, and this car has a box shape, the object detector is unsure whether it is indeed a car or a bus, which often has box shape. Consequently, even the detector detects two bounding box for this car, the score for the box is low. In contrast, image 1202 shows another image with two overlapped cars (1205 and 1206), but since both cars are still visible, the bounding boxes of both have high score. In such a case, image 1201 is more challenging and requires humans to examine.”; ][0084] As a result, compared to evaluating the classification score or size/location of individual bounding boxes, the active learning system of some embodiments consider the distribution of the detected bounding boxes that jointly considers multiple properties of the multiple bounding boxes in an image. These properties can include 1) the number of bounding boxes, 2) the sizes of the bounding boxes, 3) the classified classes of the bounding boxes, 4) the scores of the classified classes, i.e., the confidence of classification, and 5) the degree of overlapping among these bounding boxes. After the object detection is applied to an image, these properties indicate the difficulty of an image for the classification and consequently the usefulness of the image for human labelling. [0092] Namely, the scores S indicate the size of objects with low score object in this image, and the counters C indicate the degree of overlapping among the objects. We can further aggregate the scores S into a histogram called H.sub.1, which counts the number of pixels of different ranges of confidence scores. This histogram is called as a coverage histogram. If an image contains multiple bounding boxes or one big bounding box with low scores, the coverage histogram H.sub.1 will have high counts in the bins of low score ranges.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI and calculating degree overlapping between objects using a neural network as seen in OGAWA with adding a numerical label to each of the plurality of target subject as seen in Lee because this modification would achieve the expected benefits of providing quick information and easily recognizing high or low score.
Thus, the combination of RYOHEI, OGAWA and Lee teaches an information processing method implemented in a computer, comprising: obtaining a two-dimensional simulation image that is formed when a plurality of target subjects present in a three-dimensional simulation space is captured by a virtual camera; and generating, for use as training data for a machine learning model, a second simulation image that visually displays information indicating a degree of overlapping of the plurality of target subjects in the two-dimensional simulation image by adding a numerical label to each of the plurality of target subjects, the numerical label indicating the degree of overlapping
Regarding claim 2, RYOHEI, OGAWA and Lee teach the information processing method according to claim 1, wherein the generating includes calculating, based on position coordinates of each of the plurality of target subjects present in the three-dimensional simulation space and based on position coordinates of the virtual camera, the information indicating the degree of overlapping for each of the plurality of target subjects in the two-dimensional simulation image (see at least [0034] of RYOHEI “ The scale parameters o1_sx, o1_sy, and o1_sz represent the magnification ratio of the object in each axis direction in the object coordinate system. The object coordinate system refers to a coordinate system whose origin is the center of the object. [0035] The rotation angle parameters o1_θx, o1_θy, and o1_θz indicate the rotation angles of the object in each axis direction in the object coordinate system. [0036] The position coordinate parameters o1_ox, o1_oy, o1_oz indicate the center position of the object in the world coordinate system. The world coordinate system is a coordinate system whose origin is a predetermined viewpoint position.”; [0057], [0098], [0103] of OGAWA For example, a similarity L between a past candidate c.sub.1 and a current candidate c.sub.2 is calculated as follows. Here, BB denotes a vector that includes four different variables (center coordinate value x, center coordinate value y, width, and height) of each candidate BB, and f denotes the feature of each candidate. The feature refers to a feature on which each candidate is positioned, extracted from a feature map acquired from the CNN. W.sub.1 and W.sub.2 are empirically acquired coefficients and W.sub.1 > 0 and W.sub.2 > 0. More specifically, the similarity becomes higher with closer feature quantities, and the similarity becomes higher with closer detection positions and closer sizes of detection regions. [Equation 1]; Fig.9. Fig 11-12 of Lee) In addition, the same motivation is used as the rejection for claim 1.
Regarding independent claim 7, RYOHEI teaches a non-transitory computer-readable storage medium having stored therein a program that causes a computer ([0020] The CPU 11 reads and executes a program stored in the ROM 15 to control the GPU 12 , main memory 13 , VRAM 14 , ROM 15 , and display controller 16 , thereby fulfilling the function of an image generation unit 111 . The image generation unit 111 is, for example, a program (software), and is stored in the ROM 15 before the image generation device 10 is started up. When the image generating device 10 is started up, the CPU 11 reads out this program from the ROM 15 and executes it. The image generating unit 111 may be configured by dedicated hardware instead of a program) to execute a process comprising: remaining limitations of claim 7 is similar in scope to claim 1, and therefore rejected under the same rationale.
Regarding claim 8, RYOHEI, OGAWA and Lee teach the non-transitory computer-readable storage medium of claim 7, remaining limitations of claim 8 is similar in scope to claim 2, and therefore rejected under the same rationale.
Regarding claim 13, RYOHEI teaches an information processing system ([0001] The present invention relates to an image generating device, an image generating program, and an image display system having an image generating device.”) comprising: circuitry configured to ([0020] “The CPU 11 reads and executes a program stored in the ROM 15 to control the GPU 12 , main memory 13 , VRAM 14 , ROM 15 , and display controller 16 , thereby fulfilling the function of an image generation unit 111 . The image generation unit 111 is, for example, a program (software), and is stored in the ROM 15 before the image generation device 10 is started up. When the image generating device 10 is started up, the CPU 11 reads out this program from the ROM 15 and executes it. The image generating unit 111 may be configured by dedicated hardware instead of a program”) remaining limitations of claim 13 is similar in scope to claim 1, and therefore rejected under the same rationale.
Regarding claim 14, RYOHEI, OGAWA and Lee teach the information processing system of claim 13, remaining limitations of claim 14 is similar in scope to claim 2, and therefore rejected under the same rationale.
2. Claims 3, 9, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over AKITA RYOHEI, JP2013084181-English translated, (“RYOHEI”) in view of OGAWA, U.S Patent Application Publication No.20230154016 (“OGAWA”) further in view of Lee, U.S Patent Application Publication No.20200012894 (“Lee”) further in view of Gorski et al., IDS, WO 2020214699 (“Gorski”) further in view of Quan et al, IDS, U.S Patent Application Publication No.2023/0102365 (“Quan”)
Regarding claim 3, RYOHEI, OGAWA and Lee teach the information processing method according to claim 1, further comprising: training the machine learning model ( [0061] of OGAWA “Now, a method for training the learned model (specifically the CNN) for estimating an object position in an image is described. The learned model used herein is assumed to have learned an object classification task (e.g., a task for detecting a person and not detecting an animal) to some extent, so that the model learns to be able to recognize an individual based on an external feature of a predetermined object. This enables tracking of a specific object. ; [0080],[0083-0084] of Lee) In addition, the same motivation is used as the rejection for claim 1. RYOHEI, OGAWA and Lee are understood to be silent on the remaining limitations of claim 3.
In the same field of endeavor, Gorski teaches training the machine learning model by input the second simulation image to the machine learning model ([0094] Additional details relate to systems for generating simulated animal data and model are disclosed in II.S. Tat, No. 62/897,064 file September 6, 2019; the entire disclosure of which is hereby incorporated by reference. The present invention is not limited to the type of one or more statistical models of artificial intelligence techniques utilized (e. g, machine learning models, deep learning techniques). Given that the present invention is not limited by any particular application for using simulated data, such data can be used as a baseline or input to test, change, and/or modify one or more sensors, algorithms, outputs, and/or hypotheses, Moreover, data generated from one or more simulations can be used for a wide array of use cases including as a control set for identifying issues/patterns in real data, as an input in further simulations, or as an input to Artificial intelligence or machine learning models as test sets, training sets, or sets with identifiable patterns. This artificial data can be used to run simulation scenarios,: the use eases of which can range from training to improving performance and the like. For example, an artificial data set created based on real animal data from a particular athlete can be modified using the speculation system to introduce one or more deviations in the data corresponding to characteristics like fatigue or rapid heart rate changes. With this modified data, one or more simulations can be run to see how an individual (e.g., the athlete, the solider the patient) will perform in, as an example, high-stress situations or in certain environmental conditions (e.g., high altitude, high on-court temperature). This could be particularly useful in fitness applications, insurance applications, and the like, In the ease of a human (e.g., athlete) or other animal, with the system establishing the patterns between biological metrics (e. g, heart. rate, respiration, location data, biomechanical data) arid the likelihood of an occurrence happening (e.g., winning a particular match, maintaining biological functions at a certain or specified level), the speculation system can calculate One of more probabilities of certain conditional scenarios (e.g. “what-if” scenarios an likely outcomes). As an example, the system creating the artificial data can be operable to run multiple simulations in real-time or near real time for any given event (e. g,, tennis match) that may be occurring live at any given time, using n number of data inputs in the one or more simulations. Based on the results of those simulations, the system can assign a probability to a given outcome occurring. For example, if the desired analysis is “Will Player A’s HR reach 200 in current match,” the system can create a probability of this outcome happening by running one or more simulations, which may include any number of scenarios (e. g,, Player B wins the first set and Player A starts feeling stress, fatigue, and muscle tightness in specific areas of the body; air temperature and humidity increase during the match by « degrees and impacts Player A), There can be n number of such simulation scenarios an additionally simulation scenarios may be created on the fly (be , dynamically) via the speculation system's ML/A engine based on, for example, past similar matches. Once the simulations are run the output is collected and analyzed, the system may be set up to provide one or more probabilities related to the outcome under study. In a variation, more than one simulation may occur during the course of any event, with a different output (e.g., probability) resulting based on changes to the one or more input or factors (e.g., time). For example, a system that runs one or more simulations to provide Player A with a «% chance of winning a match may run one or more future simulations at a future time (e.g., 10 seconds after the first simulation, 5 minutes after the first simulation, ! hour after the first simulation, etc.) that may provide a different probability (e, g„ a simulation that is run 30 seconds after the firs simulation and utilize“ core’' and “stress” as a portion of the one or more inputs may result in a revised 52% chance for Player A to win the match because Player A lost a game within that time period and has a higher than normal stress level, which; has been shown to cause a deerease in performance in previous matches),.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI, OGAWA, Lee with inputting simulation image to a machine learning model as seen Gorski because this modification would more accurately generate a predictive indicator to predict one or more outcomes (¶0089 of Gorski) RYOHEI, OGAWA, Lee and Gorski are understood to be silent on the remaining limitations of claim 3.
In the same field of endeavor, Quan teaches wherein training includes training the machine learning model to output correct-solution data that is generated based on parameter information used in generating the two-dimensional simulation image, or to output information corresponding to the correct-solution data derived from the parameter information(see Fig. 1; ¶0074-0080 “Step 4, determining the preferred habitat of the target fish from the potential habitats by utilizing a preference learning model based on a density accumulation method, which specifically comprises: [0075] Step 4.1, acquiring a preference value of the i.sup.th species of target fish for each potential habitat by utilizing the preference learning model based on the density accumulation method; and determining the preference learning model based on the density accumulation method according to a sixth formula, wherein the sixth formula is[0076] In the formula, P.sub.ij represents the accumulated density that the i.sup.th species of target fish appears in a j.sup.th potential habitat; and Σ.sub.j=1.sup.kP.sub.j represents the total accumulated density that the target fish appears in all the potential habitats; and [0077] setting the numerical value of P.sub.ij as the preference value of the i.sup.th species of target fish for each potential habitat; [0078] Step 4.2, determining the preferred habitat of the target fish according to the preference value: [0079] comparing the preference value and a preset preference threshold, and determining the potential habitat corresponding to the preference value which is more than the preset preference threshold as the preferred habitat of the target fish; and [0080] after the preferred habitat is obtained, restoring the preferred habitat of the target fish, so as to realize targeted restoration, so that the cost is reduced, the best restoration effect is achieved, and the living environment of the fish is ensured.” where describes a machine learning model ( see step 4) is used to output correct-solution information (preferred habitat) generated based on parameter information)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI ,OGAWA, Lee and inputting simulation image to a machine learning model of Gorski with including a machine learning model is used to output correct-solution information generated based on parameter information as seen in Quan because this modification would determine the preferred habitat of the target fish from the potential habitats (¶0074 of Quan)
Thus, the combination of RYOHEI, OGAWA, Lee, Gorski and Quan teaches further comprising: training the machine learning model by input the second simulation image to the machine learning model, wherein training includes training the machine learning model to output correct-solution data that is generated based on parameter information used in generating the two-dimensional simulation image, or to output information corresponding to the correct-solution data derived from the parameter information.
Regarding claim 9, RYOHEI, OGAWA, Lee teach the non-transitory computer-readable storage medium of claim 7, wherein the process includes remaining limitations of claim 9 is similar in scope to claim 3, and therefore rejected under the same rationale.
Regarding claim 15, RYOHEI, OGAWA, Lee teach the information processing system of claim 13, wherein the circuitry is configured to remaining limitations of claim 15 is similar in scope to claim 3, and therefore rejected under the same rationale.
Regarding claim 20, RYOHEI, OGAWA, Lee, Gorski and Quan teach the information processing method according to claim 3, wherein the second simulation image further includes a two-dimensional bounding box for each of the plurality of target subjects, and the correct-solution data includes the degree of overlapping indicated by the numerical label and position information indicated by the two-dimensional bounding box (see at least [0009-0010], [0015] of RYOHEI; [0070] , [0085-0087]of OGAWA” A flowchart of the learning processing will be described with reference to FIG. 14. In S1500, the GT acquisition unit 1400 acquires the GT information and, based on the GT information, acquires the correct-answer position (the BB to be subjected to tracking) of the tracking target object in the template image and the correct-answer position of the tracking target in the search range image. In S1501, the template image acquisition unit 1401 acquires the template image. For example, the template image acquisition unit 1401 acquires an image as illustrated in FIG. 15A. In FIG. 15A, an object 1601 is the tracking target, a partial image 1602 indicates the BB of the tracking target acquired by the GT acquisition unit 1400, and a partial image 1603 indicates the region to be clipped as a template. In other words, in this case, the template image acquisition unit 1401 acquires the partial image 1603 as the template image.”; see Figs 11-12; [0095] of Lee “FIG. 14 shows another example of determining one or combination of a coverage histogram and an overlapping histogram to select the input image for labelling according to one embodiment. In this example, an input image (1401) includes two highly overlapped bounding boxes in the bottom left (1402). One bounding box correctly classify the object as a car, which has high confidence score, but the other bounding box classify the object as a bus, which has low confidence score. As a result, both histograms H.sub.1 (1403) and H (1404) show high counter values in the leftmost bin (1405 and 1406). In contrast, in the previous example of FIG. 13, the histogram H (1305) has much smaller counts than H.sub.1 (1306) since the degree of overlapping is low.”; ¶0074-0080 of Quan). In addition, the same motivation is used as the rejection for claim 3.
3. Claims 4, 6, 10, 12, 16, 18 are rejected under 35 U.S.C. 103 as being unpatentable over AKITA RYOHEI, JP2013084181-English translated, (“RYOHEI”) in view of OGAWA, U.S Patent Application Publication No.20230154016 (“OGAWA”) further in view of Lee, U.S Patent Application Publication No.20200012894 (“Lee”) further in view of HOSSLER et al, IDS, WO 2019232247 (“HOSSLER”)
Regarding claim 4, RYOHEI, OGAWA and Lee teach the information processing method according to claim 1, further comprising: RYOHEI, OGAWA and Lee are understood to be silent on the remaining limitations of claim 4.
In the same field of endeavor, HOSSLER teaches estimating, a trained machine learning model, information related to the plurality of target subjects from a taken image in which the plurality of target subjects is captured (¶0041 “One challenge with a stereo-vision system is the accurate estimation of biomass from the two-dimensional stereo camera images. For example, a single lateral dimension of a fish such as fork length not be sufficient to accurately predict the weight of the fish because of variances in fish size and feeding regimes. In some embodiments, to improve the accuracy of the weight prediction, system 100 automatically detects and captures a set of one or more morphological lateral body dimensions of a fish that are useful for accurately predicting the weight of the fish.” [0052] “Image processing system 112 may automatically detect and identify one or more or all of the landmark points and the landmark areas discussed above for purposes of predicting the weight of the fish. Image processing system 112 may do this even though the yaw, roll, and pitch angle of fish 102 captured in the stereo images may be greater than zero degrees with respect to a fish that is perfectly lateral with cameras 106. By doing so, image processing system 112 can estimate biomass from stereo images of freely swimming fish 102 in net pen 104. System 100 does not require a tube or a channel in net pen 104 through which fish 102 must swim in order to accurately estimate biomass.” ¶0075] The images of fish on which a convolutional neural network is trained may include images that are representative of images containing fish from which landmark point and landmark areas can be identified. For example, a convolutional neural network may be trained on images of fish that provide a full lateral view of the fish including the head and tail at various different yaw, roll, and pitch angles and at different sizes in the image representing different distances from cameras 106. Such training data may also be generated synthetically using a computer graphics application (e.g., a video gaming engine or a computer graphical animation application (e.g., Blender) in order to generate sufficient training data.”; ¶0030] The autonomous winch control system may adjust the location of cameras 106 according to a series of predefined or pre-programmed adjustments and /or according to detected signals in net pen 104 that indicate better or more optimal locations for capturing images of fish 102 relative to a current position and/or orientation of cameras 106. A variety of signals may be used such as, for example, machine learning and computer vision techniques applied to images captured by cameras 106 to detect schools or clusters of fish currently distant from cameras 106 such that a location that is closer to the school or cluster can be determined and the location, tilt, and / or pan of cameras 106 adjusted to capture more suitable images of the fish. The same techniques may be used to automatically determine that cameras 106 should remain or linger in a current location and /or orientation because cameras 106 are currently in a good position to capture suitable images of fish 102 for biomass estimation or other purposes.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI, OGAWA and Lee with applying a machine learning to images captures to detect schools of fish as seen in HOSSLER because this modification would accurately estimate biomass (¶0052 of HOSSLER)
Regarding claim 6, RYOHEI, OGAWA and Lee teach the information processing method according to claim 1, RYOHEI, OGAWA and Lee are understood to be silent on the remaining limitations of claim 6.
In the same field of endeavor, HOSSLER teaches wherein the target subject is a fish, and the plurality of target subjects are a plurality of fish included in a school of fish (0052] “Image processing system 112 may automatically detect and identify one or more or all of the landmark points and the landmark areas discussed above for purposes of predicting the weight of the fish. Image processing system 112 may do this even though the yaw, roll, and pitch angle of fish 102 captured in the stereo images may be greater than zero degrees with respect to a fish that is perfectly lateral with cameras 106. By doing so, image processing system 112 can estimate biomass from stereo images of freely swimming fish 102 in net pen 104. System 100 does not require a tube or a channel in net pen 104 through which fish 102 must swim in order to accurately estimate biomass.” ¶0075] The images of fish on which a convolutional neural network is trained may include images that are representative of images containing fish from which landmark point and landmark areas can be identified. For example, a convolutional neural network may be trained on images of fish that provide a full lateral view of the fish including the head and tail at various different yaw, roll, and pitch angles and at different sizes in the image representing different distances from cameras 106. Such training data may also be generated synthetically using a computer graphics application (e.g., a video gaming engine or a computer graphical animation application (e.g., Blender) in order to generate sufficient training data.”; ¶0030] The autonomous winch control system may adjust the location of cameras 106 according to a series of predefined or pre-programmed adjustments and /or according to detected signals in net pen 104 that indicate better or more optimal locations for capturing images of fish 102 relative to a current position and/or orientation of cameras 106. A variety of signals may be used such as, for example, machine learning and computer vision techniques applied to images captured by cameras 106 to detect schools or clusters of fish currently distant from cameras 106 such that a location that is closer to the school or cluster can be determined and the location, tilt, and / or pan of cameras 106 adjusted to capture more suitable images of the fish. The same techniques may be used to automatically determine that cameras 106 should remain or linger in a current location and /or orientation because cameras 106 are currently in a good position to capture suitable images of fish 102 for biomass estimation or other purposes.”) In addition, the same motivation is used as the rejection for claim 4.
Regarding claim 10, RYOHEI, OGAWA and Lee teach the non-transitory computer-readable storage medium of claim 7, wherein the process further comprises: Remaining limitations of claim 10 is similar in scope to claim 4, and therefore rejected under the same rationale.
Regarding claim 12, RYOHEI, OGAWA and Lee teach the non-transitory computer-readable storage medium of claim 7, remaining limitations of claim 12 is similar in scope to claim 6, and therefore rejected under the same rationale.
Regarding claim 16, RYOHEI, OGAWA and Lee teach the information processing system of claim 13, wherein the circuitry is configured to remaining limitations of claim 16 is similar in scope to claim 4, and therefore rejected under the same rationale.
Regarding claim 18, RYOHEI, OGAWA and Lee teach the information processing system of claim 13, remaining limitations of claim 18 is similar in scope to claim 6, and therefore rejected under the same rationale.
4. Claims 5, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over AKITA RYOHEI, JP2013084181-English translated, (“RYOHEI”) in view of OGAWA, U.S Patent Application Publication No.20230154016 (“OGAWA”) further in view of Lee, U.S Patent Application Publication No.20200012894 (“Lee”) further in view of HOSSLER et al, IDS, WO 2019232247 (“HOSSLER”) further in view of Nishikawa et al, IDS, U.S Patent Application Publication No 20230389530 (“Nishikawa”)
Regarding claim 5, RYOHEI, OGAWA, Lee and HOSSLER teach the information processing method according to claim 4, wherein the estimating includes estimating, as the information related to the plurality of target subjects, the plurality of target subjects captured in the taken image (¶0041 “One challenge with a stereo-vision system is the accurate estimation of biomass from the two-dimensional stereo camera images. For example, a single lateral dimension of a fish such as fork length not be sufficient to accurately predict the weight of the fish because of variances in fish size and feeding regimes. In some embodiments, to improve the accuracy of the weight prediction, system 100 automatically detects and captures a set of one or more morphological lateral body dimensions of a fish that are useful for accurately predicting the weight of the fish.” [0052] “Image processing system 112 may automatically detect and identify one or more or all of the landmark points and the landmark areas discussed above for purposes of predicting the weight of the fish. Image processing system 112 may do this even though the yaw, roll, and pitch angle of fish 102 captured in the stereo images may be greater than zero degrees with respect to a fish that is perfectly lateral with cameras 106. By doing so, image processing system 112 can estimate biomass from stereo images of freely swimming fish 102 in net pen 104. System 100 does not require a tube or a channel in net pen 104 through which fish 102 must swim in order to accurately estimate biomass.” ¶0075] The images of fish on which a convolutional neural network is trained may include images that are representative of images containing fish from which landmark point and landmark areas can be identified. For example, a convolutional neural network may be trained on images of fish that provide a full lateral view of the fish including the head and tail at various different yaw, roll, and pitch angles and at different sizes in the image representing different distances from cameras 106. Such training data may also be generated synthetically using a computer graphics application (e.g., a video gaming engine or a computer graphical animation application (e.g., Blender) in order to generate sufficient training data.”; ¶0030] The autonomous winch control system may adjust the location of cameras 106 according to a series of predefined or pre-programmed adjustments and /or according to detected signals in net pen 104 that indicate better or more optimal locations for capturing images of fish 102 relative to a current position and/or orientation of cameras 106. A variety of signals may be used such as, for example, machine learning and computer vision techniques applied to images captured by cameras 106 to detect schools or clusters of fish currently distant from cameras 106 such that a location that is closer to the school or cluster can be determined and the location, tilt, and / or pan of cameras 106 adjusted to capture more suitable images of the fish. The same techniques may be used to automatically determine that cameras 106 should remain or linger in a current location and /or orientation because cameras 106 are currently in a good position to capture suitable images of fish 102 for biomass estimation or other purposes.”) In addition, the same motivation is used as the rejection for claim 4. RYOHEI, OGAWA ,Lee and HOSSLER are understood to be silent on the remaining limitations of claim 5.
In the same field of endeavor, Nishikawa teaches wherein the estimating includes estimating, as the information related to the plurality of target subjects, a count of the plurality of target subjects captured ((¶0034] of Nishikawa” The fish count calculation apparatus 100 transmits sound waves within a predetermined range (an underwater space such as a pen) where a plurality of fish exist, and acquires an echo image based on the sound waves received upon being reflected by the plurality of fish. Sound waves include ultrasonic waves. Also, the fish count calculation apparatus 100 constructs an estimator that estimates the number of fish based on the echo image through machine learning using, as training data, a set of an echo image and the number of fish present within the predetermined range. Furthermore, the fish count calculation apparatus 100 uses the constructed estimator to calculate the number of fish present in an actual pen or the like based on an echo image obtained from a fish finder or the like installed near the pen or the like. Also, the fish count calculation apparatus 100 simulates the behavior of fish present within a predetermined range and echo images obtained from a fish finder or the like, and generates training echo images for machine learning.”; [0067] In step S202, the processor 101 uses a deep learning model for machine learning to construct an estimator that estimates the number of fish based on the simulation echo image (echo image) using, as training data, a training data set including the simulation echo image and the number of fish obtained in step S201. Any model may be used as the deep learning model used here. The processor 101 stores the constructed estimator for estimating the number of fish in the storage device 103. For construction of the estimator, deep learning using neural networks, multiple regression analysis, a technique using a learning space such as Look Up Table, or the like can be used. Methods other than machine learning may also be used when constructing the estimator. By using simulation echo images as training data, a larger amount of training data can be prepared compared to using actual echo images. By using a large amount of training data, an estimator with higher performance can be constructed. The simulation echo images and echo images used here are examples of training echo images.”; [0149] The fish count calculation apparatus 100 calculates the behavior (change in position over time) of fish present in a predetermined range (an underwater space such as a pen) through numerical simulation, based on the number of fish present in the predetermined range, the sizes of the fish, the size of the predetermined range, the forces acting on the fish, the flow speed of seawater, and the like. Also, the fish count calculation apparatus 100 generates a simulation echo image that simulates an echo image generated by a fish finder based on the behavior of fish in a predetermined range calculated through numerical simulation. The fish count calculation apparatus 100 calculates the behavior of the fish while changing the number of fish and the sizes of the fish, and generates simulation echo images for various numbers and sizes of fish. The fish count calculation apparatus 100 constructs an estimator that estimates the number of fish and the size of fish based on an echo image, using, as training data, a set of a simulation echo image, and the number and sizes of the fish. According to the fish count calculation apparatus 100, the behavior of the fish is calculated through numerical simulation for various numbers of fish, and the like, and by generating more simulation echo images, it is possible to create a greater amount of training data for echo images compared to when only echo images generated by an actual fish finder are used. According to the fish count calculation apparatus 100, by constructing an estimator using a larger amount of teacher data, it is possible to more accurately calculate the number of fish and the sizes of the fish (the number of fish for each class of fish size, etc.) in a pen with an unknown number of fish.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method generating an image for two-dimensional display of each of the objects projected onto a two-dimensional surface from the predetermined viewpoint position of RYOHEI, OGAWA and Lee and applying a machine learning to images captures to detect schools of fish as seen in HOSSLER with using a deep learning model for machine learning to construct an estimator for fish as seen in Nishikawa because this modification would estimates the number of fish based on the simulation echo image (echo image) using, as training data, a training data set including the simulation echo image and the number of fish obtained ([0067] of Nishikawa)
Thus, the combination of RYOHEI, OGAWA, Lee, HOSSLER and Nishikawa teaches wherein the estimating includes estimating, as information related to the plurality of target subjects, a count of the plurality of target subjects captured in the taken image.
Regarding claim 11, RYOHEI, OGAWA, Lee and HOSSLER teach the non-transitory computer-readable storage medium of claim 10, remaining limitations of claim 11 is similar in scope to claim 5, and therefore rejected under the same rationale.
Regarding claim 17, RYOHEI, OGAWA, Lee and HOSSLER teach the information processing system of claim 16, remaining limitations of claim 17 is similar in scope to claim 5, and therefore rejected under the same rationale.
Allowable Subject Matter
Claim 19 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614
Prosecution Insights