Skip to content

Wonder if Kalman filter fits imputing with lots of missing #519

Open
@XiaoJia849

Description

@XiaoJia849

i use Kalman filter to impute missing value in the time series, in the pic, the origin curve is the original curve, the red zone is missing, and i use UnscentedKalmanFilter to do the imputation, which generates the blue parts.

image

Obviously, in small missing area, the filter fits great, while in the big missing area, the imputation seems work bad. I wonder if Kalman filter fits imputing with lots of missing, or maybe i change some hyperparameters can fix this problem.

Here are my codes, part of them are copied from filterpy.kalman.UKF.UnscentedKalmanFilter. Thank you very very very very very much !!!!!!

def init_filter(fx,hx,initial_x):
    dt = 0.2
    # create sigma points to use in the filter. This is standard for Gaussian processes
    points = MerweScaledSigmaPoints(3, alpha=.1, beta=2., kappa=0)
    kf = UnscentedKalmanFilter(dim_x=3, dim_z=1, dt=dt, fx=fx, hx=hx, points=points)
    kf.x = np.array([initial_x, 0,0]) # initial state
    kf.P *= 0.2 # initial uncertainty
    z_std = 0.1 
    kf.R = np.diag([z_std**2]) 
    kf.Q = Q_discrete_white_noise(dim=3, dt=dt, var=0.01**2, block_size=1)
    return kf

def fx(x, dt):
    F = np.array([[1, dt,.5*dt*dt],
                  [0, 1,dt],
                  [0.,0.,1]], dtype=float)
    return np.dot(F, x)
def hx(x):
    # measurement function - convert state into a measurement
    # where measurements are [x_pos, y_pos]
    return np.array([x[0]])
    

def process_one_flux(fx,hx,input_flux):
    input_flux=input_flux.copy()
    idxs=np.argwhere(np.isnan(input_flux)).flatten()
    diff_idxs=np.diff(idxs)
    start_i,_=idxs[np.argmax(diff_idxs)]+1,idxs[np.argmax(diff_idxs)+1]
    initial_x=input_flux[start_i]
    kf=init_filter(fx,hx,initial_x)
    outs=[] 
    zs=input_flux.reshape((-1,1))[start_i:].tolist()
    for z in zs:
        if np.isnan(z):
            kf.predict()
            kf.update([kf.x[0]])
            outs.append(kf.x[0])
        else:
            kf.predict()
            kf.update(z)
            outs.append(kf.x[0])
    outs=np.array([np.nan]*(start_i)+outs)
    mask=np.isnan(input_flux)
    input_flux[mask]=outs[mask]
    if start_i>0:
        initial_x=input_flux[-1]
        # print("initial_x",initial_x)
        kf=init_filter(fx,hx,initial_x)
        outs=[] 
        zs=input_flux.reshape((-1,1)).tolist()
        for z in zs[::-1]:
            if np.isnan(z):
                kf.predict()
                # kf.update([kf.x[0]])
                outs.append(kf.x[0])
            else:
                kf.predict()
                kf.update(z)
                outs.append(kf.x[0])
        outs=np.array(outs[::-1])
        mask=np.isnan(input_flux)
        input_flux[mask]=outs[mask]
    return input_flux
    

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions