Python ValueError: non-broadcastable output operand with shape (352,1) doesn't match the broadcast shape (352,2)

[leeh]

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
import numpy as np
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# List of GIDs to be used
gid_list = range(367)  # Use GIDs from 0 to 366

# Lists to store predictions for each GID
all_train_predictions = []
all_test_predictions = []

for gid in gid_list:
    # Extract data for the specific GID
    gid_data = traindf_june[traindf_june['gid'] == gid][['ALIGHT_DEMAND', 'RIDE_DEMAND']]  # ALIGHT_DEMAND as input data, RIDE_DEMAND as output data

    # Time series data preprocessing function
    def create_dataset(dataset, look_back=1):
        dataX, dataY = [], []
        for i in range(len(dataset)-look_back):
            a = dataset[i:(i+look_back), 0]  # ALIGHT_DEMAND as input data
            dataX.append(a)
            dataY.append(dataset[i + look_back, 1])  # RIDE_DEMAND as output data
        return np.array(dataX), np.array(dataY)

    # Data normalization
    scaler = MinMaxScaler(feature_range=(0, 1))
    dataset = scaler.fit_transform(gid_data[['ALIGHT_DEMAND', 'RIDE_DEMAND']].values)

    # Create dataset
    look_back = 21
    X, y = create_dataset(dataset, look_back)

    # Reshape the dataset
    X = X.reshape((X.shape[0], X.shape[1], 1))

    # Split the dataset
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    # Create LSTM model
    model = Sequential()
    model.add(LSTM(50, input_shape=(look_back, 1)))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')

    # Train the model
    model.fit(X_train, y_train, epochs=5, batch_size=1, verbose=2)

    # Make predictions
    train_predict = model.predict(X_train)
    test_predict = model.predict(X_test)

    print('train_predict.shape:', train_predict.shape)
    print('test_predict.shape:', test_predict.shape)
    print('y_train.shape:', y_train.shape)
    print('y_test.shape:', y_test.shape)

    # Inverse normalization
    # train_predict = scaler.inverse_transform(train_predict)
    # print('train_predict.shape:', train_predict.shape)
    y_train = scaler.inverse_transform(y_train.reshape(-1, 1))
    # print('y_train.shape:', y_train.shape)

    # test_predict = scaler.inverse_transform(test_predict)
    # print('test_predict.shape:', test_predict.shape)
    y_test = scaler.inverse_transform(y_test.reshape(-1, 1))
    # print('y_test.shape:', y_test.shape)

    # Store predictions for each GID in lists
    all_train_predictions.append(train_predict)
    all_test_predictions.append(test_predict)

    # Visualize predictions for training and testing data
    plt.plot(np.arange(look_back, len(train_predict)+look_back), train_predict, color='red', label=f'Train Predictions (GID {gid})')
    plt.plot(np.arange(len(train_predict)+look_back, len(train_predict)+look_back+len(test_predict)), test_predict, color='blue', label=f'Test Predictions (GID {gid})')
    plt.plot(gid_data.index[look_back:], gid_data['RIDE_DEMAND'].values[look_back:], color='green', label=f'Actual RIDE_DEMAND (GID {gid})')
    plt.title(f'LSTM Model for RIDE_DEMAND Prediction (GID {gid})')
    plt.xlabel('Time')
    plt.ylabel('RIDE_DEMAND')
    plt.legend()
    plt.show()

# Convert the lists storing predictions for each GID to NumPy arrays
all_train_predictions = np.concatenate(all_train_predictions, axis=0)
all_test_predictions = np.concatenate(all_test_predictions, axis=0)

For each gid, we want to learn the LSTM model that predicts ride_demand through light_demand and output the predicted value as follows.
But in the process of reverse normalization

ValueError                                Traceback (most recent call last)
<ipython-input-25-d9740d824717> in <cell line: 16>()
     62     # train_predict = scaler.inverse_transform(train_predict)
     63     # print('train_predict.shape :',train_predict.shape)
---> 64     y_train = scaler.inverse_transform(y_train.reshape(-1, 1))
     65     # print('y_train.shape :',y_train.shape)
     66

/usr/local/lib/python3.10/dist-packages/sklearn/preprocessing/_data.py in inverse_transform(self, X)
    539         )
    540
--> 541         X -= self.min_
    542         X /= self.scale_
    543         return X

ValueError: non-broadcastable output operand with shape (352,1) doesn't match the broadcast shape (352,2)

The error occurs. How do you solve it?

 

[Chaos]

The error you're encountering, ValueError: non-broadcastable output operand with shape (352,1) doesn't match the broadcast shape (352,2), arises because the shape of your y_train variable after reshaping doesn't match the shape expected by the inverse_transform method of MinMaxScaler.

The issue is likely due to the reshaping of y_train before applying inverse_transform. You're reshaping it to have a single column, but the MinMaxScaler expects the input to have the same shape as the original input that was used for fitting the scaler, which is (n_samples, n_features).

To fix the issue, you should reshape y_train to its original shape before applying inverse_transform. Here's how you can adjust your code:

# Reshape y_train to its original shape before applying inverse_transform
y_train = y_train.reshape(-1, 1)

# Inverse normalization
y_train = scaler.inverse_transform(y_train)

You don't need to reshape y_test in this context because it's already in the correct shape (n_samples, 1).

Make sure to apply the same procedure for test_predict if you intend to inverse normalize it as well. This should resolve the ValueError you're encountering.