Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments on pages 9-10 filed on Mar 16, 2026 with respect to the motivation to combine the references in the 35 USC 103 rejection of claim 1 in the previous office action has been considered and is persuasive.
Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in this office action with respect to claim 1. Additionally, it does not appear that the title was amended to address the objection concerning lack of descriptiveness.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: “Image Processing of Feature Vectors to Reconstruct Point Cloud Images”.
Allowable Subject Matter
Claims 7-8, 10, and 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2 and 12-14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wan (NPL Doc, “Automatic Registration for Panoramic Images and Mobile LiDAR Data Based on Phase Hybrid Geometry Index Features”).
As per claim 1, Wan teaches the claimed:
1. A processor-implemented method with image processing, the method comprising:
obtaining point cloud data of a target scene (Wan in section 1, 2nd paragraph “Laser scanners are used to acquire 3D point cloud data … can easily obtain the LiDAR point cloud, image and orientation”. Figure 1 on the top right portion shows a target scene of the obtained point cloud data using a LiDAR scanner);
generating a feature map of the point cloud data by extracting a feature from the point cloud data (Wan in figure 1 on the top right portion, the claimed “generating a feature map” corresponds to the “Get intensity map” step. The figure shows that this intensity map is derived from the point cloud data from the LiDAR laser scan data. Also, please see towards the top of page 12 in item 1 for more details and in equation (26));
for each of a plurality of objects included in the target scene, generating a feature vector indicating the object in the target scene based on the feature map (Please see Wan in figure 1 in step (2) where first corresponding feature points are extracted for a plurality of objects include in the target scene. Then features vectors are created that connect the feature points between the panorama image data and mobile LiDAR data. Also, please see on page 10 in equation (19) and the text below equation (19) which recites: “For Equation (18), a 1×Nvector P can be used to indicate whether each pair of matching points is correct”. Also, figure 6 shows displacement vectors (also may be a feature vector) to connect the feature points between the panorama image data and mobile LiDAR data); and
reconstructing a panorama of the target scene based on the feature vectors of the objects (This is taught in Wan in figure 1 in step (4) the “reconstructing a panorama of the target scene” corresponds to the “Registration of the Panaroma Image” (e.g. registration between the panorama image data and LiDAR data). In addition, figure 1 shows that this reconstruction (or registration) is based on the feature vectors of the objects from step (2). This registration is a reconstruction of a panorama in the screen because it is reconstructing a panorama where the image data from the panorama is accurately registered the LiDAR data of that same scene (e.g. as mentioned in the abstract of Wan)).
As per claim 2, Wan teaches the claimed:
2. The method of claim 1, wherein, for each of the objects, the feature vector
of the object comprises a semantic feature related to semantics of the object and a geometric feature related to geometry of the object (Please see on page 10 in figure 6 where the displacement vectors (feature vectors) comprises matching of the object (or objects) and local neighborhood topology (semantics of the object) and geometry of the object (point locations)).
As per claim 12, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale. The system of Wan would have to have some type of non-transitory computer readable storage medium present in order to function and run on a computer as described by the reference.
As per claims 13 and 14, these claims are similar in scope to limitations recited in claims 1 and 2, respectively, and thus are rejected under the same rationale. The system of Wan would have to have some type of electronic device present with at least some type of processor in order to function and run on a computer as described by the reference.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3-5, 9, 11-15, 19-20 rejected under 35 U.S.C. 103 as being unpatentable over Wan in view of Kipf (Pub No. US 20220383628 A1).
As per claim 3, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed:
3. The method of claim 1, wherein, for each of the objects, the generating of the feature vector of the object indicating the object included in the target scene based on the feature map comprises:
obtaining an initial feature vector of the object; (Want teaches taking input features points from LiDAR. Wan 3. “Methodology”: “This paper proposes an automatic registration method for panoramic images and
mobile LiDAR using GPS/IMU data, which includes four steps: (1) Transform the mobile
LiDAR data into an intensity map according to the GPS/IMU; (2) Use a novel structural
feature descriptor called HGIFP and a robust mismatch elimination algorithm to
obtain corresponding feature points; (3) Iteratively obtain the optimal solution using a
particle swarm optimization (PSO) algorithm; (4) Combine the results obtained in step
(3) into new POS data, and repeat steps (1)–(3) twice to complete the registration of the
panoramic image and the mobile LiDAR data. Figure 1 shows a flowchart of the steps of the
proposed method.” It would be obvious to store these as vectors. Likewise, Kipf teaches finding those points based on a sequence of frames. Kipf [0003]: “In a first example embodiment, a method may include obtaining a first plurality of feature vectors that represent contents of a first input frame of an input data sequence and a second plurality of feature vectors that represent contents of a second input frame of the input data sequence. The second input frame may be subsequent to the first input frame. The method may also include generating, based on the first plurality of feature vectors and by a slot vector model, a first plurality of slot vectors.” The first plurality of feature vectors comprises the vector for the initial object. Kipf teaches input data through LiDAR. Kipf [0063]: “Input data 302 may represent various types of data, including, for example, image data (e.g., red-green-blue image data or gray scale image data), depth image data, point cloud data, audio data, time series data, and/or text data, among other possibilities. In some cases, input data 302 may be captured and/or generated by one or more sensors, such as visible light cameras (e.g., camera 104), near-infrared cameras (e.g., infrared camera 114), thermal cameras, stereoscopic cameras, time-of-flight (ToF) cameras, light detection and ranging (LIDAR) devices, radio detection and ranging (RADAR) devices, and/or microphones, among other possibilities. In other cases, input data 302 may additionally or alternatively include data generated by one or more users (e.g., words, sentences, paragraphs, and/or documents) or computing devices (e.g., rendered three-dimensional environments, time series plots), among other possibilities.”).
and obtaining the feature vector of the object by processing the feature map and the initial feature vector of the object by using a neural network, (Wan teaches taking feature points and extracting them through diffusion. Wan 3.1.1: “There are apparent nonlinear radiation differences (NRD) between multimodal images,
which increases the difficulty of feature point detection. As a nonlinear filter, anisotropic
filtering can achieve the smooth operation of the image and preserve the edge and texture
information of the image. Therefore, a local feature map with good edges and corner
features can be obtained by subtracting the to-be-registered and the smoothed images.
Meanwhile, phase consistency (PC) has been proven to detect feature points [40], and the edges and corner features of the image can be accurately obtained by using the maximum and minimum moments [41]. Therefore, our operation steps are as follows: (1) Smooth
the image using optimized anisotropic filtering [42] to complete the anisotropic image
diffusion; (2) Subtract the smoothed image from the to-be-registered image to obtain a
local feature map that takes anisotropy into account; (3) Construct a weighted moment
Equation and generate an anisotropic weighted moment map to extract feature points. The
anisotropic diffusion Equation is:
…
is the diffusion coefficient; k is the contrast factor; r is the gradient operator; D represents
the Laplacian operator; I is the current two-dimensional image; L is the difference image;
Remote Sens. 2022, 14, 4783 6 of 25
m means the dimension of the image; Al(Ln) represents the diffusion coefficient matrix
of the image in each dimension; Ln+1 is the result after diffusion; and _ represents the
diffusion time step, t = tn+1 tn. When the diffusion calculation is calculated, tn = 1/2s2
represents the scale, and n is the number of layers in the scale space.” Wan teaches analyzing signals in the image through convolution. Wan 3.1.2: “However, if the number of phase indices increases by increasing the convolution direction, it will increase the time complexity. Therefore, we propose a method to construct HGIFP feature descriptors based on the mixed maximum index map and orientation of the phase congruency index map (MIM-OPCIM), as shown in Figure 3d; the extraction effect is shown in Figure 3f.
Remote Sens. 2022, 14, x FOR PEER REVIEW 8 of 26
mixed maximum index map and orientation of the phase congruency index map (MIMOPCIM), as shown Figure 3d; the extraction.” Similarly, Kipf teaches the feature maps showing the features of the images, including those of the initial vector. Kipf [0022]: “These convolutional feature maps may be considered a distributed representation of the entities in the image because the features represented by the feature maps are related to different portions along the image area, but are not directly/explicitly associated with any of the entities represented in the image data. On the other hand, an object-centric representation may associate one or more features with individual entities represented in the image data. Thus, for example, each feature in a distributed representation may be associated with a corresponding portion of the perceptual representation, while each feature in an entity-centric representation may be associated with a corresponding entity contained in the perceptual representation.” The encoder model processes the frames to get the feature vectors. Kipf [0102]: “Encoder model 508 may be configured to generate feature vectors 510 based on an input frame of input data sequence 502. For example, encoder model 508 may be configured to generate, for each respective input frame of input frames 504-506, a corresponding instance of feature vectors 510. For example, feature vectors 510 may be expressed as h.sub.t=f(x.sub.t)ϵ
PNG
media_image1.png
298
298
media_image1.png
Greyscale
.sup.N×D.sup.enc, where f() may include trainable parameters of slot vector sequence system 500. In some cases, feature vectors 510 may collectively form and/or be arranged into a feature map. In some cases, feature vectors 510 may represent feature vectors 304-306 discussed in connection with FIG. 3. Thus, input matrix X may represent the feature vector h.sub.tfor a particular time step (i.e., h.sub.t=X.sub.t), where I=D.sub.enc.” This includes the initial feature vector from the original frame. The encoder model is a neural network. Kipf [0064]: “Input data 302 may be processed by way of one or more machine learning models (e.g., by an encoder model) to generate feature vectors 304-306. … In some cases, the one or more machine learning models used to process input data 302 may include convolutional neural networks. Accordingly, feature vectors 304-306 may represent a map of convolutional features of input data 302, and may thus include the outputs of various convolutional filters.”).
wherein the initial feature vector comprises an initial semantic feature related to semantics of the object and an initial geometric feature related to geometry of the object. (Kipf figs. 6-7 show geometric shapes, so the features of the geometry of the objects will be represented. As described above, the feature vectors may include semantic features.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the identification of initial feature point vectors based on point cloud data captured by Lidar entered into a network as taught by Kipf with the system of Wan in order take the geometric features of object and analyze them using a network for machine learning.
As per claim 4, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed:
4. The method of claim 3, wherein the neural network is a transformer network comprising one or more sub-neural networks comprising any one or any combination of any two or more of a cross attention layer, a self-attention layer, and a feed forward neural network layer. (Wan teaches processing an image with analysis using convolutional layers. Wan 3.1.2: “MIM is constructed via the log-Gabor convolution sequence. The convolution sequence was obtained in the PC map calculation stage, so the computational complexity of MIM is very small and does not entail too many additional calculations. The specific construction method of MIM is shown in Figure 4. Given an image 𝐼(𝑥,𝑦), 𝑀𝑒𝑛𝑜 and 𝑀𝑜𝑛𝑜 represent the even-symmetric and odd-symmetric log-Gabor wavelets at scale 𝑛 and orientation 𝑜. The two wavelet functions are convolved with the image signal to obtain the response components 𝑒𝑛𝑜(𝑥,𝑦) and 𝑜𝑛𝑜(𝑥,𝑦). For the obtained response component 𝑒𝑛𝑜(𝑥,𝑦) and 𝑜𝑛𝑜(𝑥,𝑦), we first calculate the wavelet transform amplitude 𝐴𝑛𝑜(𝑥,𝑦) at scale 𝑛 and orientation 𝑜; then, for orientation 𝑜, all magnitudes of the scale 𝑆 are summed to obtain the log-Gabor convolutional layer ” Likewise, Kipf teaches the use of a feed-forward neural network. This is used to transform the images by updating slot vectors, which represent the scenes. Kipf [0081]: “Update matrix 342 may be provided as input to neural network memory unit 320, which may be configured to update slot vectors 322-324 based on the previous values of slot vectors 322-324 (or predicted slot vectors generated based on slot vectors 322-324) and update matrix 342. Neural network memory unit 320 may include a gated recurrent unit (GRU) and/or a long-short term memory (LSTM) network, as well as other neural network or machine learning-based memory units configured to store and/or update slot vectors 322-324. For example, in addition to a GRU and/or an LSTM, neural network memory unit 320 may include one or more feed-forward neural network layers configured to further modify the values of slot vectors 322-324 after modification by the GRU and/or LSTM (and prior to being provided to task-specific machine learning model 330)”
…For each pixel, the method based on the MIM-OPCIM has a better description than that
based on the RIFT because the description dimension increases from 6 to N. Therefore, the
HGIFP feature descriptor constructed based on the MIM-OPCIM is more reliable than the
feature descriptor constructed by the RIFT. After obtaining the MIM-OPCIM, we generate
feature descriptors for each feature point in the same way as the RIFT to obtain HGIFP
feature descriptors. We build a local image patch of size J _ J pixels centered on each
feature point and assign weights to each pixel using a Gaussian function with a standard
deviation of size J/2. For the sub-grid in each local pixel block, the size is set to r _ r,
r > 6. This way includes as much neighboring information as possible. Figure 5 shows the
construction process of the HGIFP feature descriptor.” Kipf is used to analyze wave data. Kipf [0029]: “Although the slot attention model may be trained for a specific task, the architecture of the slot attention model is not task-specific and thus allows the slot attention model to be used for various tasks. The slot attention model may be used for both supervised and unsupervised training tasks. Additionally, the slot attention model does not assume, expect, or depend on the feature vectors representing a particular type of data (e.g., image data, waveform data, text data, etc.). Thus, the slot attention model may be used with any type of data that can be represented by one or more feature vectors, and the type of data may be based on the task for which the slot attention model is used.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use feed forward network as taught by Kipf with the system of Wan in order analyze the vector data using machine learning with that network structure.
As per claim 5, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed:
5. The method of claim 1, wherein the reconstructing of the panorama of the target scene based on the feature vectors of the objects comprises, for each of the objects:
obtaining a semantic label of the object, (Kipf [0023]: “Accordingly, the slot attention model may be configured to generate a plurality of entity-centric representations, referred to herein as slot vectors, based on a plurality of distributed representations, referred to herein as feature vectors. Each slot vector may be an entity-specific semantic embedding that represents the attributes or properties of one or more corresponding entities. The slot attention model may thus be considered to be an interface between perceptual representations and a structured set of variables represented by the slot vectors.” The semantic embeddings are the semantics labels.).
a mask of the object, (Kipf [0091]: “Slot vector values 402C, 404C, 406C, and 408C may represent the output of a third iteration (3×) of slot attention model 300. The visualizations of slot vector values 402A, 404A, 406A, 408A, 402B, 404B, 406B, 408B, 402C, 404C, 406C, 408C may represent visualizations of attention masks based on attention matrix 340 at each iteration and/or visualizations of reconstruction masks generated by task-specific machine learning model 330, among other possibilities.”).
a shape-related latent variable of the object, (Kipf [0062]: “322-324 based on input data 302. Feature vectors 304-306 may represent a distributed representation of the entities in input data 302, while slot vectors 322-324 may represent an entity-centric representation of these entities. Slot attention model 300 and the components thereof may represent a combination of hardware and/or software components configured to implement the functions described herein. Slot vectors 322-324 may collectively define latent representation of input data 302. In some cases, the latent representation may represent a compression of the information contained in input data 302. Thus, in some implementations, slot attention model 300 may be used as and/or viewed as a machine learning encoder. Accordingly, slot attention model 300 may be used for image reconstruction, text translation, and/or other applications that utilize machine learning encoders. Unlike certain other latent representations, each slot vector of this latent representation may capture the properties of corresponding one or more entities in input data 302, and may do so without relying on assumption about an order in which the entities are described by input data 302” The compression and representation of the data of a slot vector includes a latent variable.).
and pose information of the object (Each feature vector includes position embedding, which is the pose information. These are used for the reconstruction. Kipf [0065]: “Each respective feature vector of feature vectors 304-306 may include a position embedding that indicates a portion of input data. 302 represented by the respective feature vector. Feature vectors 304-306 may be determined, for example, by adding the position embedding to the convolutional features extracted from input data 302. Encoding the position associated with each respective feature vector of feature vectors 304-306 as part of the respective feature vector, rather than by way of the order in which the respective feature vector is provided to slot attention model 300, allows feature vectors 304-306 to be provided to slot attention model 300 in a plurality of different orders. Thus, including the position embeddings as part of feature vectors 304-306 enables slot vectors 322-324 generated by slot attention model 300 to be permutation invariant with respect to feature vectors 304-306.”).
by inputting the feature vector of the object to one or more neural networks, respectively; (Kipf [0021]: “A slot attention model may be configured to determine entity-centric (e.g., object-centric) representations of entities contained in a perceptual representation on the basis of a distributed representation of the perceptual representation. An example perceptual representation may take the form of an image that contains therein one or more entities such as objects, surfaces, regions, backgrounds, or other environmental features. M
achine learning models may be configured to generate the distributed representation of the image. For example, one or more convolutional neural networks may be configured to process the image and generate one or more convolutional feature maps, which may represent the output of various feature filters implemented by the one or more convolutional neural networks.” The neural networks process the feature vectors.).
and reconstructing the panorama of the target scene based on the semantic label of the object, the shape-related latent variable of the object, and the pose information of the object, wherein the semantic label of the object is obtained based on the semantic feature of the feature vector. (Kipf teaches using the above-mentioned aspects of the feature vectors to reconstruct the image. The slot vectors taught have the different characteristics above. Kipf [0062]: “Slot attention model 300 may be configured to generate slot vectors 322-324 based on input data 302. Feature vectors 304-306 may represent a distributed representation of the entities in input data 302, while slot vectors 322-324 may represent an entity-centric representation of these entities. Slot attention model 300 and the components thereof may represent a combination of hardware and/or software components configured to implement the functions described herein. Slot vectors 322-324 may collectively define latent representation of input data 302. In some cases, the latent representation may represent a compression of the information contained in input data 302. Thus, in some implementations, slot attention model 300 may be used as and/or viewed as a machine learning encoder. Accordingly, slot attention model 300 may be used for image reconstruction, text translation, and/or other applications that utilize machine learning encoders. Unlike certain other latent representations, each slot vector of this latent representation may capture the properties of corresponding one or more entities in input data 302, and may do so without relying on assumption about an order in which the entities are described by input data 302.” Kipf mentions reconstruction and uses multiple images, Kipf does not explicitly mention reconstruction a panorama per se. As mentioned above for claim 1, Wan in figure 1 in step (4) the “reconstructing a panorama of the target scene” corresponds to the “Registration of the Panaroma Image” (e.g. registration between the panorama image data and LiDAR data). In addition, figure 1 shows that this reconstruction (or registration) is based on the feature vectors of the objects from step (2). This registration is a reconstruction of a panorama in the screen because it is reconstructing a panorama where the image data from the panorama is accurately registered the LiDAR data of that same scene (e.g. as mentioned in the abstract of Wan). The images contain semantic information that could be used for the vectors that reconstruct the image. Wan Abstract: “Firstly, a novel feature descriptor called a hybrid geometric structure index of phase (HGIFP) is
built to capture the structural information of the images. Then, a set of corresponding feature points is obtained from the two images using the constructed feature descriptor combined with a robust false-match elimination algorithm. The average pixel distance of the corresponding feature points is used as the error function. Finally, in order to complete the accurate registration of the mobile LiDAR data and panoramic images and improve computational efficiency, we propose the assumption of local motion invariance of 3D–2D corresponding feature points and minimize the error function
through multiple reprojections to achieve the best registration parameters.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use reconstruction of image feature vectors in specific formats into a panorama as taught by Kipf with the system of Wan to represent the features of the images of Wan as feature vectors and input them into the neural networks of Kipf for processing and reconstruction into a panorama.
As per claim 9, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed
9. The method of claim 1, further comprising: determining whether the target scene is a new scene;
in response to determining that the target scene is a new scene, storing the feature vector of the object; (Kipf teaches that the scenes are based on slot vectors. Thus, a current scene and future scene is used for a slot vector. A future scene will eventually be a current scene.
Kipf [0033]: “Specifically, the predictor model may be trained to model the dynamics and/or interactions of the entities represented by the plurality of slot vectors, and thus forecast the state of these entities in a subsequent input frame (e.g., at a future time). For example, in the context of video, the predictor model may be configured to predict, based on a current state of objects in a scene (as represented by the slot vectors), a future state of the objects in the scene (as represented by the predicted slot vectors). The slot vector model may also be configured to process the predicted slot vectors generated for a particular (e.g., future) input frame, along with feature vectors corresponding to the particular input frame, and, based thereon, generate another plurality of slot vectors corresponding to the particular input frame.”).
and in response to determining that the target scene is a scene associated with a previous scene, obtaining an initial feature vector of an object corresponding to the target scene by initializing a feature vector of the object corresponding to the target scene by using a feature vector of an object included in the previous scene. (Kipf [0081]: “Update matrix 342 may be provided as input to neural network memory unit 320, which may be configured to update slot vectors 322-324 based on the previous values of slot vectors 322-324 (or predicted slot vectors generated based on slot vectors 322-324) and update matrix 342. Neural network memory unit 320 may include a gated recurrent unit (GRU) and/or a long-short term memory (LSTM) network, as well as other neural network or machine learning-based memory units configured to store and/or update slot vectors 322-324. For example, in addition to a GRU and/or an LSTM, neural network memory unit 320 may include one or more feed-forward neural network layers configured to further modify the values of slot vectors 322-324 after modification by the GRU and/or LSTM (and prior to being provided to task-specific machine learning model 330).” The slot vectors describe the scenes. An updated slot vector is a current scene and is based on the values of a previous slot vector, which is a previous scene. This will initialize a new feature vector, and the scene is associated with a previous scene to be updates, since some of the elements of the vector are incorporated in the new scene.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the initial feature vectors of an object through different concurrent scenes taught by Kipf with the system of Wan to represent movement of an object represented by point clouds in the image analysis of Wan and output it as a panorama.
As per claim 11, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed
11. The method of claim 5, wherein the reconstructing of the panorama of the target scene comprises:
obtaining an object grid of the object by decoding the latent variable using a decoder; (Wan top of pg. 9: “For each pixel, the method based on the MIM-OPCIM has a better description than that based on the RIFT because the description dimension increases from 6 to N. Therefore, the HGIFP feature descriptor constructed based on the MIM-OPCIM is more reliable than the feature descriptor constructed by the RIFT. After obtaining the MIM-OPCIM, we generate feature descriptors for each feature point in the same way as the RIFT to obtain HGIFP feature descriptors. We build a local image patch of size J _ J pixels centered on each feature point and assign weights to each pixel using a Gaussian function with a standard deviation of size J/2. For the sub-grid in each local pixel block, the size is set to r _ r, r > 6. This way includes as much neighboring information as possible. Figure 5 shows the construction process of the HGIFP feature descriptor.”
Similarly, Kip teaches breaking an image down into pixels and colors and using a grid. A latent representation is breaking features down into vectors for analysis. Kipf [0116]: “In implementations where input data sequence 502 includes image data, task-specific model 530 may individually decode each of slot vectors 514 and/or predicted slot vectors 520 using a spatial broadcast decoder. Specifically, each slot may be broadcast onto a two-dimensional grid which may be augmented with position embeddings. Each grid may be decoded using a convolutional neural network (the parameters of which may be shared across each slot vector of slot vectors 514) to generate an output of size W×H×4, where W and H represent the width and height, respectively, of the reconstructed slot-specific image data and the additional 4 dimensions represent the red, green, and blue color The alpha masks may be normalized across the slot-specific images using a softmax function and may be used as mixture weights to recombine and/or mix the slot-specific images into a final reconstruction of the original image frame. In other examples, the slot decoder model may be and/or may include aspects of patch-based decoders.” The slot vectors can include latent representations which are decoded. Kipf [0062]: “Slot vectors 322-324 may collectively define latent representation of input data 302. In some cases, the latent representation may represent a compression of the information contained in input data 302. Thus, in some implementations, slot attention model 300 may be used as and/or viewed as a machine learning encoder. Accordingly, slot attention model 300 may be used for image reconstruction, text translation, and/or other applications that utilize machine learning encoders. Unlike certain other latent representations, each slot vector of this latent representation may capture the properties of corresponding one or more entities in input data 302, and may do so without relying on assumption about an order in which the entities are described by input data 302.”).
and obtaining a panoramic view of the target scene by combining the semantic label of the object, the mask of the object, the object grid of the object, and the pose information of the object. (All of these aspects of the objects are used to reconstruct a panorama of a scene as described in the rejection to claim 5.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the breaking down of an image vector into a grid representation taught by Kipf with the system of Wan represent the image more clearly for panoramic reconstruction.
As per claim 12, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf teaches the claimed:
12. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1. (Kipf [0004]: “In a second example embodiment, a system may include a processor and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations. The operations may include obtaining a first plurality of feature vectors that represent contents of a first input frame of an input data sequence and a second plurality of feature vectors that represent contents of a second input frame of the input data sequence.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the non-transitory computer-readable medium as taught by Kipf with the system of Wan in order to use that medium to implement the instructions for the process.
As per claim 13, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale. The use of a neural network operating on image data requires computation which is done on an electronic device.
As per claim 14, this claim is similar in scope to limitations recited in claim 3, and thus is rejected under the same rationale.
As per claim 15, this claim is similar in scope to limitations recited in claim 5, and thus is rejected under the same rationale.
As per claim 19, this claim is similar in scope to limitations recited in claim 9, and thus is rejected under the same rationale.
As per claim 20, this claim is similar in scope to limitations recited in claim 11, and thus is rejected under the same rationale.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Wan in view of Kipf and further in view of Selviah (Pub No. US 20230146134 A1).
As per claim 6, Wan alone does not explicitly teach the claimed limitations.
However, Wan in combination with Kipf and Selviah teaches the claimed:
6. The method of claim 5, wherein the reconstructing of the panorama of the target scene comprises:
obtaining subdivided pose information of the object as pose information of the object based on the mask of the object, the pose information of the object, and the point cloud data; (Selviah teaches subdivided features of the vectors that depict 3D objects. Selviah Claim 2: “The method according to claim 1, wherein the domain 3D vector field comprises a plurality of sub-fields, and the comparing comprises comparing each of the plurality of sub-fields with the object 3D vector field of each of the plurality of candidate objects; wherein the comparing includes dividing the 3D representation of the domain into a plurality of sub-divisions, and each sub-field corresponds to a respective one of the sub-divisions; or wherein the sub fields are individual or groups of features extracted from the domain 3D vector field by a segmentation algorithm.” Selviah teaches that the vectors of the objects are sub divided by segmentation. The vector data being subdivided is based off point cloud representations of 3D objects. Selviah [0166]: “Point cloud representations of the relevant portion of the 3D representation of the domain, and of the candidate object, are obtained. The relevant portion of the 3D representation being either the whole of the 3D representation of the domain (in a case in which the comparing at S104is performed on the domain 3D vector field representing the whole of the domain without division into sub divisions), or, in a case in which the comparing comprises dividing the 3D representation of the domain into a plurality of sub-divisions, the sub-division corresponding to the sub-field based on which the candidate object is determined to be present in the imaged domain.”).obtaining a subdivided shape-related latent variable of the object as a shape-related latent variable of the object based on the mask of the object, the shape-related latent variable of the object, and the point cloud data; (The vector subdivision taught by Selviah is applied to point cloud data represented by feature data. The feature vectors can include the latent variable and the masks described above in Kipf. Kipf [0091]:“Slot vector values 402C, 404C, 406C, and 408C may represent the output of a third iteration (3×) of slot attention model 300. The visualizations of slot vector values 402A, 404A, 406A, 408A, 402B, 404B, 406B, 408B, 402C, 404C, 406C, 408C may represent visualizations of attention masks based on attention matrix 340 at each iteration and/or visualizations of reconstruction masks generated by task-specific machine learning model 330, among other possibilities.” The slot vectors are the scenes with the object and position data. They also include latent representation. Kipf [0062]: “Slot attention model 300 and the components thereof may represent a combination of hardware and/or software components configured to implement the functions described herein. Slot vectors 322-324 may collectively define latent representation of input data 302. In some cases, the latent representation may represent a compression of the information contained in input data 302. Thus, in some implementations, slot attention model 300 may be used as and/or viewed as a machine learning encoder. Accordingly, slot attention model 300 may be used for image reconstruction, text translation, and/or other applications that utilize machine learning encoders. Unlike certain other latent representations, each slot vector of this latent representation may capture the properties of corresponding one or more entities in input data 302, and may do so without relying on assumption about an order in which the entities are described by input data 302.”).
and reconstructing the panorama of the target scene based on the semantic label of the object, the mask of the object, and the shape-related latent variable of the object, and the pose information of the object. (Kipf teaches reconstructing the object based on these different aspects of the object vectors, as described above.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the subdivisions of position vectors based on point cloud data as taught by Selviah with the system of Wan and Kipf in order to subdivide the vectors of the objects and organize the data contained in those vectors for more clear processing.
As per claim 16, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS JOHN FOSTER whose telephone number is (571)272-5053. The examiner can normally be reached Mon, Fri 8:30-6. Tues-Thurs 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS JOHN FOSTER/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights